A camera motion estimation method and system based on optical flow-guided foreground masking

Abstract

This invention discloses a camera motion estimation method and system based on optical flow-guided foreground masking, comprising: dividing video frames into image sequences and dividing them into uniform grids; estimating the optical flow value of each pixel in the image using the RAFT algorithm to obtain motion information at each grid vertex; randomly sampling the motion information of a finite number of points in the uniform grid space, and extracting the background optical flow field from the motion information of the sampled points using the RANSAC algorithm; calculating the difference between the optical flow of the current pixel and the background optical flow pixel by pixel based on the image frame, and normalizing it to obtain the background motion weight at the corresponding pixel, and combining it with the motion information at the vertex to generate fused motion information at each grid vertex; and performing temporal accumulation of the fused motion information at each grid vertex to generate the camera motion trajectory at each vertex. This invention can more accurately separate and estimate camera motion when processing videos containing multiple moving subjects or with parallax, and effectively obtain camera motion information.

Description

Technical Field

[0001] This invention relates to the field of video image processing technology, and more specifically to a camera motion estimation method and system based on optical flow-guided foreground masking. Background Technology

[0002] Currently, with the development of digital image processing technology, electronic video stabilization technology has gradually become mainstream. It does not require hardware equipment and achieves stable processing of video frames through software algorithms, directly estimating the motion vectors of consecutive frames. This not only significantly reduces costs but also improves the stabilization effect to a certain extent.

[0003] However, current electronic image stabilization technology still faces many challenges, such as how to effectively estimate the camera's motion path in complex and dynamically changing environments, ensure the robustness of the algorithm in various scenarios, and address the efficiency of real-time processing. In particular, how to achieve good video stabilization while ensuring real-time performance in handheld devices and embedded systems with limited computing resources remains an urgent problem to be solved.

[0004] In digital video stabilization technology, motion estimation is a crucial step, as the extracted camera motion trajectory directly affects the accuracy of subsequent motion smoothing and motion compensation. In addition, motion estimation is also a relatively time-consuming step in the digital video stabilization process.

[0005] Feature point detection is a commonly used motion estimation algorithm. This method uses a selected feature set to represent the information of the entire image, and performs feature point matching through a corresponding feature matching algorithm to calculate the relative motion vector between frames. When using feature matching to obtain motion estimation parameters, the first step is to extract salient feature points from the video frame. Depending on different image feature selection criteria, feature extraction algorithms will generate feature descriptions of different dimensions. The higher the dimension of the feature description, the greater the computational cost. The more feature points extracted, the more accurate the obtained motion parameters will be. Commonly used feature extraction algorithms include SIFT, ORB, and BRISK algorithms. Choosing the appropriate algorithm for different scenarios can improve the efficiency and accuracy of feature extraction.

[0006] However, in real life, video footage often contains a variety of complex scenes, which pose significant challenges to camera motion estimation. First, there's strong parallax variation: this phenomenon typically refers to significant changes in the relative positions of different objects within a video frame. In such scenes, the movement of foreground and background objects relative to the observer leads to significant differences in perspective, affecting the accurate estimation of camera motion. Second, there's depth-of-field variation: during video recording, adjustments to focal length, aperture, or shooting distance can alter the range of objects in sharp focus within the frame. This variation not only affects video quality but also complicates camera motion estimation. Finally, there's multi-dynamic foreground occlusion: during video recording, the movement of multiple dynamic objects in the foreground can cause occlusion between themselves or with background objects. This occlusion makes it difficult for the camera to accurately distinguish between foreground and background when calculating motion trajectories, resulting in trajectory estimation errors.

[0007] Accurate camera motion extraction becomes a complex and challenging task when faced with these difficult scenarios. This also brings additional technical challenges to subsequent video stabilization processing, forcing us to explore more advanced algorithms and methods to achieve efficient and accurate camera motion extraction in complex and ever-changing environments.

[0008] Therefore, how to efficiently and accurately estimate camera motion in real time in complex and ever-changing environments is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0009] In view of this, the present invention provides a camera motion estimation method based on optical flow-guided foreground masking to solve some of the technical problems mentioned in the background art.

[0010] To achieve the above objectives, the present invention adopts the following technical solution:

[0011] A camera motion estimation method based on optical flow-guided foreground masking includes the following steps:

[0012] S1. Divide the video frames into an image sequence, and then divide the image sequence into a uniform grid;

[0013] S2. The RAFT algorithm is used to estimate the optical flow value of each pixel in the image, and the motion information at each grid vertex is further obtained;

[0014] S3. Randomly sample motion information of a finite number of points in a uniform grid space, and extract the background optical flow field from the motion information of the sampled points using the RANSAC algorithm;

[0015] S4. Calculate the difference between the optical flow value of the current pixel and the background optical flow field on a pixel-by-pixel basis of the image frame, and normalize the difference value to obtain the background motion weight at the corresponding pixel. Combine the obtained motion information at the vertex to generate the fused motion information at each grid vertex.

[0016] S5. By accumulating the fused motion information at the vertices of each grid region over time, the motion trajectory of the camera at each vertex is generated, resulting in multiple camera paths.

[0017] Preferably, in step S1, a grid-based motion estimation modeling method is used to divide the video frame into a uniform grid and calculate the motion path of each grid vertex individually.

[0018] Preferably, the specific content of step S2 is as follows:

[0019] S21. By extracting features from the input image, a hierarchical feature representation is generated;

[0020] S22. By constructing a cost volume, feature matching relationships between image pairs are quantified and modeled;

[0021] S23. Optimize the cost volume using deformable convolution, thereby iteratively improving the accuracy of optical flow estimation;

[0022] S24. Optimize the optical flow results through the refinement module to obtain accurate and stable pixel motion information.

[0023] Preferably, in step S22, the formula for constructing the cost volume is:

[0024]

[0025] Where d represents the offset within the search range, and Similarity represents the feature similarity calculation function, used to calculate the feature of each pixel (i,j) in image I1 at all possible locations (i,j) in image I2. ' ,j ' Feature similarity score;

[0026] In step S23, the specific implementation of deformable convolution is as follows:

[0027]

[0028] Among them, C ' The optimized cost volume is represented by H, which represents the parameters of the deformable convolution, including the offset and the weights of the convolution kernel. Through deformable convolution, the model can dynamically adjust the position of the convolution kernel to capture more accurate feature matching relationships.

[0029] In step S24, the specific function of the refinement module is as follows:

[0030]

[0031] Among them, u init and v init For the preliminary estimate of the optical flow vector, and This is the final optical flow result after optimization by the refined module.

[0032] Preferably, in step S3, the specific content of the background optical flow field is extracted from the motion information of the sampling points using the RANSAC algorithm as follows:

[0033] The homography transformation matrix is ​​used as the model. In each iteration, a certain number of samples are randomly selected from the sampling points and the homography transformation matrix is ​​calculated by the least squares method. The transformed points are obtained by transforming all points using the homography transformation matrix. The error of each point is calculated based on the pixel-based motion value estimated by RAFT optical flow. Points with errors less than the threshold are considered as inliers. The homography transformation matrix with the smallest sum of errors of inliers is selected as the optimal model, which is the background optical flow field motion information.

[0034] Preferably, the homography transformation matrix H is used for all points. After transformation, the predicted coordinates of the transformed points are obtained. The calculation formula is:

[0035]

[0036] Obtain predicted coordinates Then, the predicted camera motion is represented as:

[0037]

[0038] Pixel-based motion values ​​estimated by RAFT optical flow The error at each point is calculated as follows:

[0039] .

[0040] Preferably, in step S4, the fused motion information at each mesh vertex is:

[0041]

[0042] Where k is the background motion weight obtained based on the foreground mask, and v b For background optical flow field motion information obtained based on sampled optical flow information, v p This is the vertex motion information obtained based on the RAFT network.

[0043] A camera motion estimation system based on optical flow-guided foreground mask, based on the aforementioned optical flow-guided foreground mask-based camera motion estimation method, includes: a mesh generation module, an optical flow estimation module, an optical flow-guided foreground mask generation module, a motion information fusion generation module, and a camera motion path generation module;

[0044] The grid partitioning module is used to divide video frames into image sequences and then divide the image sequences into uniform grids.

[0045] The optical flow estimation module is used to estimate the optical flow value of each pixel in the image using the RAFT algorithm, and further obtain the motion information at each grid vertex;

[0046] The optical flow-guided foreground mask generation module is used to randomly sample motion information of a finite number of points in a uniform grid space and extract the background optical flow field from the motion information of the sampled points using the RANSAC algorithm.

[0047] The motion information generation module is used to calculate the difference between the optical flow value of the current pixel and the background optical flow field on a pixel-by-pixel basis of the image frame, and normalize the difference value to obtain the background motion weight at the corresponding pixel. Combined with the obtained motion information at the vertex, the module generates the fused motion information at each grid vertex.

[0048] The camera motion path generation module is used to accumulate the fused motion information at the vertices of each grid region in a temporal manner to generate the camera motion trajectory at each vertex, thus obtaining multiple bundled camera paths.

[0049] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the aforementioned camera motion estimation method based on optical flow-guided foreground masking.

[0050] A processing terminal includes a memory and a processor. The memory stores a computer program that can run on the processor. When the processor executes the computer program, it implements the aforementioned camera motion estimation method based on optical flow-guided foreground masking.

[0051] As can be seen from the above technical solutions, compared with the prior art, the present invention discloses a camera motion estimation method and system based on optical flow-guided foreground masking, which does not rely on feature point detection, ensuring that the algorithm still maintains good robustness in some texture-sparse and feature-deficient regions; it adds a foreground motion estimation module to extract more accurate camera motion information; it does not rely on neural networks, and there is no need to build, train or update models, thus meeting the requirements of real-time scenes;

[0052] Specifically, this method replaces a single camera motion path with a uniformly gridded camera path, with each location having its own camera path model. This more flexible model can effectively solve nonlinear motion caused by parallax. Furthermore, the method analyzes the complexity of camera motion and proposes a foreground mask estimation module based on optical flow guidance. This module effectively assesses the interference degree of each region on camera motion and extracts a more accurate camera motion path for each region. When processing videos containing multiple moving subjects or with parallax, it can more accurately separate and estimate camera motion, effectively acquiring camera motion information. This motion information can be better used in downstream computer vision tasks, such as video stabilization. Attached Figure Description

[0053] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0054] Figure 1 A schematic diagram of a camera motion estimation method based on optical flow-guided foreground mask provided by the present invention;

[0055] Figure 2 This is a schematic diagram illustrating the foreground mask detection performance of the present invention on the publicly available NUS dataset. Detailed Implementation

[0056] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0057] This invention discloses a camera motion estimation method based on optical flow-guided foreground masking, such as... Figure 1 This includes the following steps:

[0058] S1. Divide the video frames into an image sequence, and then divide the image sequence into a uniform grid;

[0059] S2. The RAFT algorithm is used to estimate the optical flow value of each pixel in the image, and the motion information at each grid vertex is further obtained;

[0060] S3. Randomly sample motion information of a finite number of points in a uniform grid space, and extract the background optical flow field from the motion information of the sampled points using the RANSAC algorithm;

[0061] S4. Calculate the difference between the optical flow value of the current pixel and the background optical flow field on a pixel-by-pixel basis of the image frame, and normalize the difference value to obtain the background motion weight at the corresponding pixel. Combine the obtained motion information at the vertex to generate the fused motion information at each grid vertex.

[0062] S5. By accumulating the fused motion information at the vertex of each grid region over time, the motion trajectory of the camera at each vertex is generated, resulting in multiple bundled camera paths;

[0063] The motion trajectories of different grid vertices integrate depth information and eliminate interference from moving objects in the foreground, thus ensuring that the generated motion trajectories are smoother and more accurate.

[0064] To further implement the above technical solution, in step S1, a grid-based motion estimation modeling method is adopted to divide the video frame into a 20×20 uniform grid and calculate the motion path of each grid vertex individually.

[0065] To further implement the above technical solution, in step S2, RAFT is a deep learning-based optical flow estimation algorithm. Its core idea is to infer pixel-level motion information by constructing a cyclic feature matching framework and utilizing the feature similarity between image pairs. Specifically:

[0066] S21. By extracting features from the input image, a hierarchical feature representation is generated;

[0067] In this embodiment, multi-scale features of the image are extracted using a deep network model including but not limited to ResNet-50. The output of the feature extraction network is a feature pyramid containing feature maps at different levels.

[0068] S22. By constructing a cost volume, feature matching relationships between image pairs are quantified and modeled;

[0069] In this embodiment, the cost volume is constructed by calculating the similarity between two feature maps; specifically, for each pixel (i,j) in image I1, its possible positions (i,j) in image I2 are calculated. ' ,j ' The feature similarity scores are stacked to form a three-dimensional cost volume of size H×W×D, where H and W are the height and width of the image, and D is the size of the search area; the formula for constructing the cost volume is:

[0070]

[0071] Where d represents the offset within the search range, and Similarity represents the feature similarity calculation function, used to calculate the feature of each pixel (i,j) in image I1 at all possible locations (i,j) in image I2. ' ,j ' Feature similarity score;

[0072] S23. Optimize the cost volume using deformable convolution, thereby iteratively improving the accuracy of optical flow estimation;

[0073] In this embodiment, deformable convolution can adaptively adjust the position and shape of the convolution kernel to better capture non-rigid motion and complex deformation in the image; in RAFT, deformable convolution is used to optimize cost volume and dynamically adjust the convolution operation to better adapt to different motion patterns.

[0074] The specific implementation of deformable convolution is as follows:

[0075]

[0076] Among them, C ' The optimized cost volume is represented by H, which represents the parameters of the deformable convolution, including the offset and the weights of the convolution kernel. Through deformable convolution, the model can dynamically adjust the position of the convolution kernel to capture more accurate feature matching relationships.

[0077] S24. The optical flow results are optimized through the refinement module to obtain accurate and stable pixel motion information;

[0078] In this embodiment, the thinning module includes a series of convolutional layers and upsampling operations to recover high-resolution optical flow information and reduce noise and inconsistent motion vectors. In addition, the thinning module also includes post-processing steps such as introducing optical flow smoothing and consistency checks to further improve the quality of the results.

[0079] The specific function of the refinement module is as follows:

[0080]

[0081] Among them, u init and v init For the preliminary estimate of the optical flow vector, and This is the final optical flow result after optimization by the refined module.

[0082] To further implement the above technical solution, and to eliminate areas of local motion, such as moving vehicles or objects in the shooting scene, or motion areas caused by parallax, step S3 extracts the specific content of the background optical flow field from the motion information of the sampling points using the RANSAC (Random Sample Consensus) algorithm:

[0083] Camera motion typically includes translation, rotation, and scaling. To describe the motion parameters, the homography transformation matrix is ​​used as the mathematical expression of the model in the algorithm.

[0084] In each iteration, a certain number of samples are randomly selected from the sampling points, and the homography transformation matrix is ​​calculated using the least squares method. Using homography transformation matrix For all points After transformation, the predicted coordinates of the transformed points are obtained. The calculation formula is:

[0085]

[0086] Obtain predicted coordinates Then, the predicted camera motion is represented as:

[0087]

[0088] Pixel-based motion values ​​estimated by RAFT optical flow Calculate the error for each point. Points with errors less than a threshold are considered interior points. The error calculation formula is as follows:

[0089]

[0090] The homography transformation matrix that minimizes the sum of errors at the interior points is selected as the optimal model, which represents the background optical flow field motion information.

[0091] To further implement the above technical solution, in step S4, the fused motion information at each mesh vertex is as follows:

[0092]

[0093] Where k is the background motion weight obtained based on the foreground mask, and v b For background optical flow field motion information obtained based on sampled optical flow information, v p This is the vertex motion information obtained based on the RAFT network.

[0094] In another embodiment, the foreground mask detection performance of the optical flow-guided foreground mask-based camera motion estimation method of the present invention is verified on the publicly available NUS dataset, such as... Figure 2Experimental results and analysis show that the present invention can effectively extract regions that differ from camera motion.

[0095] A camera motion estimation system based on optical flow-guided foreground masking, which is based on an optical flow-guided foreground masking method, includes: a mesh generation module, an optical flow estimation module, an optical flow-guided foreground mask generation module, a motion information fusion generation module, and a camera motion path generation module;

[0096] The grid partitioning module is used to divide video frames into image sequences and then divide the image sequences into uniform grids.

[0097] The optical flow estimation module is used to estimate the optical flow value of each pixel in the image using the RAFT algorithm, and further obtain the motion information at each grid vertex;

[0098] The optical flow-guided foreground mask generation module is used to randomly sample motion information of a finite number of points in a uniform grid space and extract the background optical flow field from the motion information of the sampled points using the RANSAC algorithm.

[0099] The motion information generation module is used to calculate the difference between the optical flow value of the current pixel and the background optical flow field on a pixel-by-pixel basis of the image frame, and normalize the difference value to obtain the background motion weight at the corresponding pixel. Combined with the obtained motion information at the vertex, the module generates the fused motion information at each grid vertex.

[0100] The camera motion path generation module is used to accumulate the fused motion information at the vertices of each grid region in a temporal manner to generate the camera motion trajectory at each vertex, thus obtaining multiple bundled camera paths.

[0101] A computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements a camera motion estimation method based on optical flow-guided foreground masking.

[0102] A processing terminal includes a memory and a processor. The memory stores a computer program that can run on the processor. When the processor executes the computer program, it implements a camera motion estimation method based on optical flow-guided foreground masking.

[0103] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0104] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A camera motion estimation method based on optical flow-guided foreground masking, characterized in that, Includes the following steps: S1. Divide the video frames into an image sequence, and then divide the image sequence into a uniform grid; S2. The RAFT algorithm is used to estimate the optical flow value of each pixel in the image, and the motion information at each grid vertex is further obtained; S3. Randomly sample motion information of a finite number of points in a uniform grid space, and extract the background optical flow field from the motion information of the sampled points using the RANSAC algorithm; S4. Calculate the difference between the optical flow value of the current pixel and the background optical flow field on a pixel-by-pixel basis of the image frame, and normalize the difference value to obtain the background motion weight at the corresponding pixel. Combine the obtained motion information at the vertex to generate the fused motion information at each grid vertex. S5. By accumulating the fused motion information at the vertex of each grid region over time, the motion trajectory of the camera at each vertex is generated, resulting in multiple bundled camera paths; In step S4, the fused motion information at each mesh vertex is as follows: Where k is the background motion weight obtained based on the foreground mask, and v b For background optical flow field motion information obtained based on sampled optical flow information, v p This is the vertex motion information obtained based on the RAFT network.

2. The camera motion estimation method based on optical flow-guided foreground masking according to claim 1, characterized in that, Step S1: Using a grid-based motion estimation modeling method, the video frame is divided into a uniform grid, and the motion path of each grid vertex is calculated individually.

3. The camera motion estimation method based on optical flow-guided foreground masking according to claim 1, characterized in that, The specific content of step S2 is as follows: S21. By extracting features from the input image, a hierarchical feature representation is generated; S22. By constructing a cost volume, feature matching relationships between image pairs are quantified and modeled; S23. Optimize the cost volume using deformable convolution, thereby iteratively improving the accuracy of optical flow estimation; S24. Optimize the optical flow results through the refinement module to obtain accurate and stable pixel motion information.

4. The camera motion estimation method based on optical flow-guided foreground masking according to claim 3, characterized in that, In step S22, the formula for constructing the cost volume is: Where d represents the offset within the search range, and Similarity represents the feature similarity calculation function, used to calculate the feature of each pixel (i,j) in image I1 at all possible locations (i,j) in image I2. ' ,j ' Feature similarity score; In step S23, the specific implementation of deformable convolution is as follows: Among them, C ' The optimized cost volume is represented by H, which represents the parameters of the deformable convolution, including the offset and the weights of the convolution kernel. Through deformable convolution, the model can dynamically adjust the position of the convolution kernel to capture more accurate feature matching relationships. In step S24, the specific function of the refinement module is as follows: Among them, u init and v init For the preliminary estimate of the optical flow vector, and This is the final optical flow result after optimization by the refined module.

5. The camera motion estimation method based on optical flow-guided foreground masking according to claim 1, characterized in that, Step S3, extracting the specific content of the background optical flow field from the motion information of the sampling points using the RANSAC algorithm: The homography transformation matrix is ​​used as the model. In each iteration, a certain number of samples are randomly selected from the sampling points and the homography transformation matrix is ​​calculated by the least squares method. The transformed points are obtained by transforming all points using the homography transformation matrix. The error of each point is calculated based on the pixel-based motion value estimated by RAFT optical flow. Points with errors less than the threshold are considered as inliers. The homography transformation matrix with the smallest sum of errors of inliers is selected as the optimal model, which is the background optical flow field motion information.

6. The camera motion estimation method based on optical flow-guided foreground masking according to claim 5, characterized in that, Use the homography transformation matrix H for all points After transformation, the predicted coordinates of the transformed points are obtained. The calculation formula is: Obtain predicted coordinates Then, the predicted camera motion is represented as: Pixel-based motion values ​​estimated by RAFT optical flow The error at each point is calculated as follows: 。 7. A camera motion estimation system based on optical flow-guided foreground masking, characterized in that, A camera motion estimation method based on optical flow-guided foreground mask according to any one of claims 1-6 includes: a mesh generation module, an optical flow estimation module, an optical flow-guided foreground mask generation module, a motion information fusion generation module, and a camera motion path generation module. The grid partitioning module is used to divide video frames into image sequences and then divide the image sequences into uniform grids. The optical flow estimation module is used to estimate the optical flow value of each pixel in the image using the RAFT algorithm, and further obtain the motion information at each grid vertex; The optical flow-guided foreground mask generation module is used to randomly sample motion information of a finite number of points in a uniform grid space and extract the background optical flow field from the motion information of the sampled points using the RANSAC algorithm. The motion information generation module is used to calculate the difference between the optical flow value of the current pixel and the background optical flow field on a pixel-by-pixel basis of the image frame, and normalize the difference value to obtain the background motion weight at the corresponding pixel. Combined with the obtained motion information at the vertex, the module generates the fused motion information at each grid vertex. The camera motion path generation module is used to accumulate the fused motion information at the vertices of each grid region in a temporal manner to generate the camera motion trajectory at each vertex, thus obtaining multiple bundled camera paths.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements a camera motion estimation method based on optical flow-guided foreground masking as described in any one of claims 1-6.

9. A processing terminal, comprising a memory and a processor, wherein the memory stores a computer program executable on the processor, characterized in that, When the processor executes the computer program, it implements a camera motion estimation method based on optical flow-guided foreground mask as described in any one of claims 1-6.

Images