Camera motion estimation method and system under foreground mask based on optical flow guidance

Abstract

The invention discloses a camera motion estimation method and system under a foreground mask based on optical flow guidance. The method comprises the following steps: dividing a video frame into an image sequence and dividing uniform grids; estimating an optical flow value of each pixel in the image by adopting an RAFT algorithm, and obtaining motion information at the vertex of each grid; motion information of finite points is randomly sampled in the uniform grid space, and a background optical flow field is extracted from the motion information of the sampling points through an RANSAC algorithm; calculating the difference between the current pixel point optical flow and the background optical flow pixel by pixel based on the image frame, normalizing to obtain the background motion weight of the corresponding pixel, and generating the fusion motion information of the vertex of each grid by combining the motion information of the vertex; performing time sequence accumulation through the fused motion information at the vertex of each grid, and generating a motion track of the camera at each vertex; when a video containing a plurality of motion subjects or having parallax is processed, the camera motion can be more accurately separated and estimated, and the motion information of the camera is effectively obtained.

Description

Technical Field

[0001] The present invention relates to the technical field of video image processing, and more particularly to a method and system for estimating camera motion under a foreground mask based on optical flow guidance. Background Art

[0002] At present, with the development of digital image processing technology, electronic video stabilization technology has gradually become mainstream. It does not require hardware equipment. It can achieve stable processing of video frames through software algorithms and directly estimate the motion vectors of consecutive frames. This not only significantly reduces costs, but also improves the stabilization effect to a certain extent.

[0003] However, current electronic image stabilization technology still faces many challenges, such as how to effectively estimate the camera's motion path in complex and dynamically changing environments, ensure the robustness of the algorithm in various scenarios, and the efficiency of real-time processing. Especially in handheld devices and embedded systems with limited computing resources, how to achieve good video stabilization while ensuring real-time performance remains an urgent problem to be solved.

[0004] In digital video stabilization technology, motion estimation is a crucial link. The extracted camera motion trajectory will directly affect the accuracy of subsequent motion smoothing and motion compensation. In addition, motion estimation is also a relatively time-consuming step in the digital video stabilization process.

[0005] Feature point detection technology is a commonly used motion estimation algorithm. This method uses a selected feature set to represent the information of the entire image, and performs feature point matching through the corresponding feature matching algorithm to calculate the relative motion vector between frames. When using the feature matching method to obtain motion estimation parameters, the first step is to extract significant feature points in the video screen. Based on different image feature selection criteria, the feature extraction algorithm will generate feature descriptions of different dimensions. The higher the dimension of the feature description, the greater the computational cost. The more feature points are extracted, the more accurate the motion parameters obtained will be. Commonly used feature extraction algorithms include SIFT algorithm, ORB algorithm and BRISK algorithm. Choosing the appropriate algorithm in different scenarios can improve the efficiency and accuracy of feature extraction.

[0006] However, in real life, the video content captured often contains a variety of complex scenes, which pose significant challenges to the estimation of camera motion. First, strong parallax changes: This phenomenon usually refers to the large changes in the relative positions of different objects in the video frame. In such scenes, the movement of foreground and background objects relative to the observer will lead to significant differences in perspective, which affects the accurate estimation of camera motion. Second, depth of field changes: During the video shooting process, due to factors such as adjusting the focal length, aperture or shooting distance, the range of clear imaging in the picture changes. This change not only affects the video quality, but also makes it difficult to estimate camera motion. Finally, the phenomenon of multiple dynamic foreground occlusion: During the video shooting process, the movement of multiple dynamic objects in the foreground may cause occlusion between them or with background objects. This occlusion phenomenon makes it difficult for the camera to accurately distinguish between the foreground and background when calculating the motion trajectory, resulting in trajectory estimation errors.

[0007] When faced with these difficult scenes, accurately extracting camera motion becomes a complex and challenging task. This also brings additional technical challenges to subsequent video stabilization processing, forcing us to explore more advanced algorithms and methods to achieve efficient and accurate camera motion extraction in complex and changing environments.

[0008] Therefore, how to efficiently and accurately estimate camera motion in real time in a complex and changing environment is a problem that those skilled in the art urgently need to solve. Summary of the Invention

[0009] In view of this, the present invention provides a camera motion estimation method under foreground mask based on optical flow guidance to solve some of the technical problems mentioned in the background technology.

[0010] In order to achieve the above object, the present invention adopts the following technical solutions:

[0011] A camera motion estimation method under foreground mask based on optical flow guidance includes the following steps:

[0012] S1. Divide the video frame into an image sequence, and divide the image sequence into a uniform grid;

[0013] S2. Use the RAFT algorithm to estimate the optical flow value of each pixel in the image and further obtain the motion information of each grid vertex;

[0014] S3. Randomly sample the motion information of a limited number of points in a uniform grid space and extract the background optical flow field from the motion information of the sampled points using the RANSAC algorithm;

[0015] S4. Calculate the difference between the optical flow value of the current pixel and the background optical flow field pixel by pixel based on the image frame, normalize the difference value to obtain the background motion weight at the corresponding pixel, and combine the obtained vertex motion information to generate the fused motion information at each mesh vertex;

[0016] S5. Generate the motion trajectory of the camera at each vertex by performing temporal accumulation of the fused motion information at each vertex of the grid area, and obtain multiple bundles of camera paths.

[0017] Preferably, in step S1, a grid-based motion estimation modeling method is used to divide the video frame into uniform grids, and the motion path of each grid vertex is calculated separately.

[0018] Preferably, the specific content of step S2 is:

[0019] S21. Generating a hierarchical feature representation by extracting features from the input image;

[0020] S22. Constructing a cost volume for quantifying and modeling the feature matching relationship between image pairs;

[0021] S23. Use deformable convolution to optimize the cost volume, thereby gradually and iteratively improving the accuracy of optical flow estimation;

[0022] S24. Optimize the optical flow results through the refinement module to obtain accurate and stable pixel motion information.

[0023] Preferably, in step S22, the cost volume is constructed using the formula:

[0024]

[0025] Where d represents the offset within the search range, Similarity represents the feature similarity calculation function, which is used to calculate the feature of each pixel (i, j) in image I1 at all possible locations (i ' ,j ' )’s feature similarity score;

[0026] In step S23, the specific implementation of deformable convolution is:

[0027]

[0028] Among them, C ' Represents the optimized cost volume, and H represents the parameters of the deformable convolution, including the offset and the weight of the convolution kernel. Through deformable convolution, the model can dynamically adjust the position of the convolution kernel to capture more accurate feature matching relationships.

[0029] In step S24, the specific role of the refinement module is as follows:

[0030]

[0031] Among them, u init and v init is the initial estimated optical flow vector, and This is the final optical flow result after optimization by the refinement module.

[0032] Preferably, in step S3, the specific content of extracting the background optical flow field from the motion information of the sampling points by using the RANSAC algorithm is:

[0033] The homography transformation matrix is ​​used as the model. In each iteration, a certain number of samples are randomly selected from the sampling points and the homography transformation matrix is ​​calculated by the least squares method. All points are transformed using the homography transformation matrix to obtain the transformed points. The error of each point is calculated according to the pixel-based motion value estimated by RAFT optical flow. Points with errors less than the threshold are considered to be inliers. The homography transformation matrix with the smallest total error of the inliers is selected as the optimal model, which is the background optical flow field motion information.

[0034] Preferably, the homography transformation matrix H is used to transform all points After transformation, the predicted coordinates of the transformed point are obtained The calculation formula is:

[0035]

[0036] Get predicted coordinates After that, the predicted camera motion is expressed as:

[0037]

[0038] Pixel-based motion values ​​estimated from RAFT optical flow Calculate the error at each point as:

[0039] .

[0040] Preferably, in step S4, the fused motion information at each mesh vertex is:

[0041]

[0042] Among them, k is the background motion weight obtained based on the foreground mask, v b is the background optical flow field motion information obtained based on the sampled optical flow information, v p is the vertex motion information obtained based on the RAFT network.

[0043] A camera motion estimation system under foreground mask based on optical flow guidance, based on the above-mentioned camera motion estimation method under foreground mask based on optical flow guidance, comprising: a grid division module, an optical flow estimation module, a foreground mask generation module based on optical flow guidance, a fused motion information generation module and a camera motion path generation module;

[0044] A grid division module, used for dividing the video frames into image sequences and dividing the image sequences into uniform grids;

[0045] The optical flow estimation module is used to estimate the optical flow value of each pixel in the image using the RAFT algorithm, and further obtain the motion information of each grid vertex;

[0046] The foreground mask generation module based on optical flow guidance is used to randomly sample the motion information of a limited number of points in a uniform grid space and extract the background optical flow field from the motion information of the sampled points through the RANSAC algorithm;

[0047] The fused motion information generation module is used to calculate the difference between the optical flow value of the current pixel and the background optical flow field based on the image frame pixel by pixel, and normalize the difference value to obtain the background motion weight at the corresponding pixel. The fused motion information at each mesh vertex is generated by combining the obtained vertex motion information;

[0048] The camera motion path generation module is used to perform temporal accumulation of the fused motion information at the vertex of each grid area, generate the motion trajectory of the camera at each vertex, and obtain multiple bundles of camera paths.

[0049] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements a method for estimating camera motion under a foreground mask based on optical flow guidance.

[0050] A processing terminal includes a memory and a processor. The memory stores a computer program that can be run on the processor. When the processor executes the computer program, the method for estimating camera motion under a foreground mask based on optical flow guidance is implemented.

[0051] As can be seen from the above technical solutions, compared with the existing technology, the present invention provides a method and system for estimating camera motion under foreground mask based on optical flow guidance. This method does not rely on feature point detection, ensuring that the algorithm maintains good robustness in some areas with sparse textures and lack of features. The foreground motion estimation module is added to extract more accurate camera motion information. The method does not rely on neural networks and does not require the construction, training, or updating of models, thus meeting the requirements of real-time scenarios.

[0052] Specifically: A camera path divided into a uniform grid is used instead of a single camera motion path. Each position has its own camera path model. This more flexible model can effectively solve the nonlinear motion caused by parallax. In addition, this method analyzes the complexity of camera motion and proposes a foreground mask estimation module based on optical flow guidance. It effectively evaluates the interference of each area on the camera motion and extracts a more accurate camera motion path for each area. When processing videos containing multiple moving subjects or with parallax, it can more accurately separate and estimate camera motion and effectively obtain camera motion information. This motion information can be better used in downstream computer vision tasks such as video stabilization. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0054] Figure 1 A schematic diagram of a camera motion estimation method under a foreground mask based on optical flow guidance provided by the present invention;

[0055] Figure 2 This is a schematic diagram of the foreground mask detection effect of the present invention on the public NUS dataset provided by the present invention. DETAILED DESCRIPTION

[0056] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0057] The embodiment of the present invention discloses a method for estimating camera motion under foreground mask based on optical flow guidance, such as Figure 1 , including the following steps:

[0058] S1. Divide the video frame into an image sequence, and divide the image sequence into a uniform grid;

[0059] S2. Use the RAFT algorithm to estimate the optical flow value of each pixel in the image and further obtain the motion information of each grid vertex;

[0060] S3. Randomly sample the motion information of a limited number of points in a uniform grid space and extract the background optical flow field from the motion information of the sampled points using the RANSAC algorithm;

[0061] S4. Calculate the difference between the optical flow value of the current pixel and the background optical flow field pixel by pixel based on the image frame, normalize the difference value to obtain the background motion weight at the corresponding pixel, and combine the obtained vertex motion information to generate the fused motion information at each mesh vertex;

[0062] S5. Perform temporal accumulation of the fused motion information at the vertex of each grid area to generate the motion trajectory of the camera at each vertex, thereby obtaining multiple bundles of camera paths;

[0063] The motion trajectories of different mesh vertices integrate depth information and eliminate the interference of foreground moving objects, thereby ensuring that the generated motion trajectories are smoother and more accurate.

[0064] In order to further implement the above technical solution, in step S1, a grid-based motion estimation modeling method is used to divide the video frame into a 20×20 uniform grid, and the motion path of each grid vertex is calculated separately.

[0065] To further implement the above technical solution, in step S2, RAFT is an optical flow estimation algorithm based on deep learning. Its core idea is to infer pixel-level motion information by using the feature similarity between image pairs by building a cyclic feature matching framework. The specific content is:

[0066] S21. Generating a hierarchical feature representation by extracting features from the input image;

[0067] In this embodiment, a deep network model including but not limited to ResNet-50 is used to extract multi-scale features of the image. The output of the feature extraction network is a feature pyramid containing feature maps at different levels.

[0068] S22. Constructing a cost volume for quantifying and modeling the feature matching relationship between image pairs;

[0069] In this embodiment, the cost volume is constructed by calculating the similarity between the two feature maps; specifically, for each pixel (i, j) in the image I1, its feature at all possible locations (i ' ,j ' ), the similarity scores are stacked to form a three-dimensional cost volume with a size of H×W×D, where H and W are the height and width of the image, and D is the size of the search range; the cost volume construction formula is:

[0070]

[0071] Where d represents the offset within the search range, Similarity represents the feature similarity calculation function, which is used to calculate the feature of each pixel (i, j) in image I1 at all possible locations (i ' ,j ' )’s feature similarity score;

[0072] S23. Use deformable convolution to optimize the cost volume, thereby gradually and iteratively improving the accuracy of optical flow estimation;

[0073] In this embodiment, deformable convolution can adaptively adjust the position and shape of the convolution kernel to better capture non-rigid motion and complex deformation in the image. In RAFT, deformable convolution is used to optimize the cost volume and dynamically adjust the convolution operation to better adapt to different motion patterns.

[0074] The specific implementation of deformable convolution is:

[0075]

[0076] Among them, C ' Represents the optimized cost volume, and H represents the parameters of the deformable convolution, including the offset and the weight of the convolution kernel. Through deformable convolution, the model can dynamically adjust the position of the convolution kernel to capture more accurate feature matching relationships.

[0077] S24. Optimize the optical flow results through the refinement module to obtain accurate and stable pixel motion information;

[0078] In this embodiment, the refinement module includes a series of convolutional layers and upsampling operations to restore high-resolution optical flow information and reduce noise and inconsistent motion vectors. In addition, the refinement module also includes post-processing steps such as optical flow smoothing and consistency checking to further improve the quality of the results.

[0079] The specific role of the refinement module is expressed as follows:

[0080]

[0081] Among them, u init and v init is the initial estimated optical flow vector, and This is the final optical flow result after optimization by the refinement module.

[0082] To further implement the above technical solution, in order to eliminate local motion areas, such as moving vehicles or objects appearing in the captured scene, and motion areas caused by parallax, step S3 uses the RANSAC (Random Sample Consensus) algorithm to extract the background optical flow field from the motion information of the sampling points. The specific content is:

[0083] Camera motion usually includes translation, rotation, and scaling. In order to describe the motion parameters, the algorithm uses the homography transformation matrix as the mathematical expression of the model.

[0084] In each iteration, a certain number of samples are randomly selected from the sampling points and the homography transformation matrix is ​​calculated by the least squares method. , using the homography transformation matrix For all points After transformation, the predicted coordinates of the transformed point are obtained The calculation formula is:

[0085]

[0086] Get predicted coordinates After that, the predicted camera motion is expressed as:

[0087]

[0088] Pixel-based motion values ​​estimated from RAFT optical flow Calculate the error of each point. Points with errors less than the threshold are considered to be internal points. The error calculation formula is:

[0089]

[0090] The homography transformation matrix with the smallest total error of the inliers is selected as the optimal model, which is the background optical flow field motion information.

[0091] In order to further implement the above technical solution, in step S4, the fused motion information at each mesh vertex is:

[0092]

[0093] Among them, k is the background motion weight obtained based on the foreground mask, v b is the background optical flow field motion information obtained based on the sampled optical flow information, v p is the vertex motion information obtained based on the RAFT network.

[0094] In another embodiment, the foreground mask detection effect of the camera motion estimation method under the foreground mask guided by optical flow of the present invention is verified on the public NUS dataset. Figure 2,The experimental results and analysis show that the ,presentation can better extract the region with differences in ,camera motion.

[0095] A camera motion estimation system under foreground mask based on optical flow guidance is based on a camera motion estimation method under foreground mask based on optical flow guidance, comprising: a grid division module, an optical flow estimation module, a foreground mask generation module based on optical flow guidance, a fusion motion information generation module and a camera motion path generation module;

[0096] A grid division module, used for dividing the video frames into image sequences and dividing the image sequences into uniform grids;

[0097] The optical flow estimation module is used to estimate the optical flow value of each pixel in the image using the RAFT algorithm, and further obtain the motion information of each grid vertex;

[0098] The foreground mask generation module based on optical flow guidance is used to randomly sample the motion information of a limited number of points in a uniform grid space and extract the background optical flow field from the motion information of the sampled points through the RANSAC algorithm;

[0099] The fused motion information generation module is used to calculate the difference between the optical flow value of the current pixel and the background optical flow field based on the image frame pixel by pixel, and normalize the difference value to obtain the background motion weight at the corresponding pixel. The fused motion information at each mesh vertex is generated by combining the obtained vertex motion information;

[0100] The camera motion path generation module is used to perform temporal accumulation of the fused motion information at the vertex of each grid area, generate the motion trajectory of the camera at each vertex, and obtain multiple bundles of camera paths.

[0101] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements a camera motion estimation method under a foreground mask based on optical flow guidance.

[0102] A processing terminal includes a memory and a processor. The memory stores a computer program that can be run on the processor. When the processor executes the computer program, a camera motion estimation method under a foreground mask guided by optical flow is implemented.

[0103] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method description.

[0104] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A camera motion estimation method under foreground mask based on optical flow guidance, characterized in that: The following steps are involved: S1. Divide the video frame into an image sequence, and divide the image sequence into a uniform grid; S2. Use the RAFT algorithm to estimate the optical flow value of each pixel in the image and further obtain the motion information of each grid vertex; S3. Randomly sample the motion information of a limited number of points in a uniform grid space and extract the background optical flow field from the motion information of the sampled points using the RANSAC algorithm; S4. Calculate the difference between the optical flow value of the current pixel and the background optical flow field pixel by pixel based on the image frame, normalize the difference value to obtain the background motion weight at the corresponding pixel, and combine the obtained vertex motion information to generate the fused motion information at each mesh vertex; S5. Generate the motion trajectory of the camera at each vertex by performing temporal accumulation of the fused motion information at each vertex of the grid area, and obtain multiple bundles of camera paths.

2. The method for estimating camera motion under foreground mask based on optical flow guidance according to claim 1, characterized in that: Step S1: Using a grid-based motion estimation modeling method, the video frame is divided into uniform grids, and the motion path of each grid vertex is calculated separately.

3. The method for estimating camera motion under foreground mask based on optical flow guidance according to claim 1, characterized in that: The specific content of step S2 is: S21. Generating a hierarchical feature representation by extracting features from the input image; S22. Constructing a cost volume for quantifying and modeling the feature matching relationship between image pairs; S23. Use deformable convolution to optimize the cost volume, thereby gradually and iteratively improving the accuracy of optical flow estimation; S24. Optimize the optical flow results through the refinement module to obtain accurate and stable pixel motion information.

4. The method for estimating camera motion under foreground mask based on optical flow guidance according to claim 3, characterized in that: In step S22, the cost volume is constructed using the formula: ； Where d represents the offset within the search range, Similarity represents the feature similarity calculation function, which is used to calculate the feature of each pixel (i, j) in image I1 at all possible locations (i ' ,j ' )’s feature similarity score; In step S23, the specific implementation of deformable convolution is: ； Among them, C ' Represents the optimized cost volume, and H represents the parameters of the deformable convolution, including the offset and the weight of the convolution kernel. Through deformable convolution, the model can dynamically adjust the position of the convolution kernel to capture more accurate feature matching relationships. In step S24, the specific role of the refinement module is as follows: ； Among them, u init and v init is the initial estimated optical flow vector, and This is the final optical flow result after optimization by the refinement module.

5. The method for estimating camera motion under foreground mask based on optical flow guidance according to claim 1, characterized in that: Step S3, extracting the background optical flow field from the motion information of the sampling points using the RANSAC algorithm. The specific content is: The homography transformation matrix is ​​used as the model. In each iteration, a certain number of samples are randomly selected from the sampling points and the homography transformation matrix is ​​calculated by the least squares method. All points are transformed using the homography transformation matrix to obtain the transformed points. The error of each point is calculated according to the pixel-based motion value estimated by RAFT optical flow. Points with errors less than the threshold are considered to be inliers. The homography transformation matrix with the smallest total error of the inliers is selected as the optimal model, which is the background optical flow field motion information.

6. The method for estimating camera motion under foreground mask based on optical flow guidance according to claim 5, characterized in that: Use the homography transformation matrix H to transform all points After transformation, the predicted coordinates of the transformed point are obtained The calculation formula is: ； Get predicted coordinates After that, the predicted camera motion is expressed as: ； Pixel-based motion values ​​estimated from RAFT optical flow Calculate the error at each point as: 。 7. The method for estimating camera motion under foreground mask based on optical flow guidance according to claim 1, characterized in that: In step S4, the fused motion information at each mesh vertex is: ； Among them, k is the background motion weight obtained based on the foreground mask, v b is the background optical flow field motion information obtained based on the sampled optical flow information, v p is the vertex motion information obtained based on the RAFT network.

8. A camera motion estimation system under foreground mask based on optical flow guidance, characterized in that: A camera motion estimation method under a foreground mask based on optical flow guidance according to any one of claims 1 to 7, comprising: a grid division module, an optical flow estimation module, a foreground mask generation module based on optical flow guidance, a fusion motion information generation module, and a camera motion path generation module; A grid division module, used for dividing the video frames into image sequences and dividing the image sequences into uniform grids; The optical flow estimation module is used to estimate the optical flow value of each pixel in the image using the RAFT algorithm, and further obtain the motion information of each grid vertex; The foreground mask generation module based on optical flow guidance is used to randomly sample the motion information of a limited number of points in a uniform grid space and extract the background optical flow field from the motion information of the sampled points through the RANSAC algorithm; The fused motion information generation module is used to calculate the difference between the optical flow value of the current pixel and the background optical flow field based on the image frame pixel by pixel, and normalize the difference value to obtain the background motion weight at the corresponding pixel. The fused motion information at each mesh vertex is generated by combining the obtained vertex motion information; The camera motion path generation module is used to perform temporal accumulation of the fused motion information at the vertex of each grid area, generate the motion trajectory of the camera at each vertex, and obtain multiple bundles of camera paths.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for estimating camera motion under a foreground mask based on optical flow guidance described in any one of claims 1 to 7 is implemented.

10. A processing terminal comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, characterized in that: When the processor executes the computer program, the method for estimating camera motion under foreground mask based on optical flow guidance is implemented as described in any one of claims 1 to 7.

Images