Motion vector estimation method, system and readable storage medium

Abstract

This invention relates to a motion vector estimation method, system, and readable storage medium. Utilizing a multi-layered pyramid structure, motion vectors are calculated from different layers. Downsampling motion vector calculation employs dense optical flow, iteratively updating layer by layer from the bottom to the top using Kalman filtering. During upsampling, the motion vector is updated using a local block matching algorithm based on the region indicated by the motion vector update results of the upsampling layer. Correction is then performed using the filtered motion vector obtained during the bottom-to-top process. This iterative update from the top to the bottom using Kalman filtering achieves clean and accurate motion vectors. This method combines the advantages of two motion vector calculation algorithms while compensating for their respective disadvantages. Through iterative processing from the bottom to the top and back to the bottom within the complete pyramid structure, it can quickly obtain clean and relatively accurate motion vectors even when processing videos with poor signal-to-noise ratios.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a motion vector estimation method, system, and readable storage medium. Background Technology

[0002] Motion vector estimation is widely used in image encoding / decoding algorithms and denoising algorithms. Its core idea is to use correlation algorithms to find the reference position of each pixel or pixel block in the previous frame of the current image. The direction of the difference between this reference position and the position of the pixel or pixel block in the current image is the direction of the motion vector; the magnitude of this difference is the magnitude of the motion vector. The ultimate goal of motion vector estimation is to find the position of the pixel or pixel block to be processed in the previous frame of the current image, and use this position as a reference to predict or denoise the pixel or pixel block to be processed in the current image. For example, in encoding algorithms such as H.264 and H.265, a reference pixel block is found as the prediction result for the current block to be processed. The residual between the prediction result and the actual result is calculated, and the residual is encoded. Finally, only the encoded motion vector result and the encoded residual result need to be stored, which can significantly improve the compression ratio while ensuring quality. In denoising algorithms, traditional temporal filtering controls the denoising intensity based on whether the current position is moving. If stationary, it averages the pixels at the same position as the reference image to improve the signal-to-noise ratio; if moving, it does not perform denoising. However, if the motion vector of the object is known, the position of the point or block to be processed in the current image can be found in the reference frame. Then, the reference pixel block or reference point at that position can be moved to the position of the block to be processed. After traversing the entire image, a deformed reference image can be obtained. Theoretically, the pixels or pixel blocks at the same position in the deformed reference image are the same as those in the current image. Therefore, temporal filtering can be performed on the current image based on the deformed reference image. Compared with traditional temporal denoising methods, using a deformed reference image can achieve the effect of denoising both stationary and moving objects simultaneously.

[0003] There are two main types of motion vector estimation methods: block matching-based motion vector estimation algorithms and optical flow-based motion vector estimation methods.

[0004] Block matching assumes that most moving objects move slowly in each frame, meaning the position of the current point in the previous frame is within a local region of the current point. By limiting the search range, all possible points within the search range are compared with the current point to find the best match. Block matching typically restricts the movement of small local regions of objects to translation, without rotation or stretching. Therefore, it calculates the similarity between blocks within the local region centered on the current point and blocks of the same size centered on each reference point within the search range. The Manhattan distance is a relatively simple method for calculating similarity. The reference point with the highest similarity is the best match. Although the similarity calculation algorithm uses the simple Manhattan distance formula, the search method is a full search. If the search range is large, the similarity of a large number of reference blocks needs to be calculated simultaneously, making real-time performance difficult. Therefore, faster block matching algorithms are needed. Faster block matching algorithms usually use a certain search strategy, such as square search, cross search, or diamond search, and iterate a few times to find a near-optimal result. In h265, motion information from the previous frame and motion information from points already processed around the current point to be processed are used to infer the predicted value of the motion vector of the current point. The position pointed to by the predicted value is used as the search center for the search, further improving the search efficiency.

[0005] Optical flow algorithms are generally divided into sparse optical flow and dense optical flow. Sparse optical flow first finds all corner points in the image, calculates the motion vectors of all corner points, and then interpolates to obtain the motion vector of each pixel in the image. Dense optical flow, on the other hand, directly solves for the motion vector of each pixel in the image. The optical flow solution for motion vectors is based on two assumptions: the illumination of the moving object remains constant at adjacent time points, and the motion vectors in local regions are identical. First, an optical flow equation is established based on the assumption that the illumination of the object remains constant at adjacent time points. Then, a system of optical flow equations is established based on the assumption that the motion vectors in local regions are identical. Next, the horizontal gradient, vertical gradient, and the difference between pixels in the current image and the reference image at the same location are calculated using the current image and a reference image. These three factors are then substituted into the system of optical flow equations, and the motion vector is solved using the least squares method.

[0006] However, neither the block matching algorithm nor the optical flow algorithm considers noise models. If the image has significant noise, both methods will obtain incorrect motion vectors, or a mixture of correct and incorrect motion vectors with an overall chaotic appearance. The pyramid method can effectively improve the signal-to-noise ratio of an image, so the block matching algorithm also utilizes an improvement to the pyramid method. This involves performing block matching on the top layer of the pyramid, then upsampling the resulting motion vectors. The position pointed to by this upsampling vector in the next layer is used as the center of the motion search in the next layer, continuing the full search in the local region, iterating until the bottom layer is reached. While this method can obtain relatively clean motion vectors, accuracy is difficult to guarantee. Firstly, the downsampling area loses high-frequency information, so the identified motion vectors may not be accurate. Secondly, each upsampling motion vector estimation assumes that the previous layer's motion vector estimate is accurate, and then searches the local region pointed to by that vector. If the previous layer's motion vector is inaccurate, subsequent results will also be inaccurate. Therefore, improvements need to be made to the block matching pyramid model to address these two problems. Summary of the Invention

[0007] The purpose of this invention is to provide a motion vector estimation method, system, and readable storage medium, which solves the problem that the accuracy of traditional motion vector estimation algorithms is poor due to noise interference in scenarios with poor signal-to-noise ratio.

[0008] To achieve the above objectives, the present invention provides a motion vector estimation method, comprising:

[0009] S1. Obtain the current image C0 to be processed and the reference image P0;

[0010] S2. Calculate the update results of the motion vector of each point in the current layer of the pyramid using the dense optical flow method. Initialize i = 1;

[0011] S3. Update results of motion vectors for the (i-1)th layer Downsampling is performed in each dimension to obtain the motion vector prediction result of the i-th layer.

[0012] S4. For the current image C in the (i-1)th layer i-1 and reference image P i-1 Perform downsampling to obtain the current image C of the i-th layer. i and reference image P i ;

[0013] S5, according to the aforementioned C i and P i The dense optical flow method is used to calculate the motion vector of each point in the current layer of the pyramid.

[0014] S6. Establish a downsampling Kalman filter model, and use the observations converted from the motion vectors to update the predicted result of the motion vectors, and obtain the updated result of the motion vectors of the i-th layer after filtering

[0015] S7. Set the total number of pyramid layers to n, let i = i + 1. If i < n, repeat steps S3 - S6. If i = n, execute step S8 to obtain the updated result of the motion vectors of the top layer of the pyramid after filtering

[0016] S8. For each pixel of the current image C at the top layer n-1 , use the updated result of the motion vectors at the top layer to point to the position of the reference image P n-1 , and perform a full-search block matching algorithm in the local area to obtain the motion vector of the block matching search as the secondary updated result of the motion vectors at the top layer

[0017] S9. Initialize j = n - 2, and upsample each dimension of the secondary updated result of the motion vectors of the (j + 1)-th layer to obtain the preliminary predicted result of the motion vectors of the j-th layer

[0018] S10. For each pixel of the current image C of the j-th layer j , use the preliminary predicted result of the motion vectors of the j-th layer to point to the position of the reference image P j , and perform a full-search block matching algorithm in the local area to obtain the motion vector of the block matching search as the final predicted result of the motion vectors of the j-th layer

[0019] S11. Establish an upsampling Kalman filter model, and use the observations converted from the updated result of the motion vectors of the j-th layer to update the final predicted result of the motion vectors of the j-th layer , and obtain the secondary updated result of the motion vectors of the j-th layer after filtering

[0020] S12. Let j = j - 1. If j ≥ 0, repeat steps S9 - S11. If j < 0, it is the final result of the motion vector estimation.

[0021] Optionally, the reference image is the image obtained by performing spatio-temporal domain noise reduction processing on the previous frame of the current image. ​

[0022] Optionally, in steps S3 and S4, the downsampling method used is 2x2 mean downsampling.

[0023] Optionally, in step S9, the upsampling method used is bilinear interpolation upsampling.

[0024] Optionally, in step S6, when establishing the downsampled Kalman filter model of the motion vector, the result of step S3 is first used as a basis. Design a prediction model, then design a measurement matrix based on the optical flow algorithm, and then calculate the Kalman gain. The results from step S5 are then used to... The transformed observations for the Update.

[0025] Optionally, in step S11, when establishing the upsampled Kalman filter model of the motion vector, the result of step S10 is first used as a basis. A prediction model is established, a measurement matrix is ​​designed based on the optical flow algorithm, and then the Kalman gain is calculated. The results of step S6 are then used to determine the final measurement model. The transformed observations update the motion vector.

[0026] Based on the same technical concept, the present invention also provides a motion vector estimation system, comprising:

[0027] The data input unit is configured to input the current image C0 to be processed and the reference image P0;

[0028] The first computational unit is configured to use dense optical flow to calculate the update results of the motion vector of each point in the current layer of the pyramid. And initialize i = 1;

[0029] The motion vector downsampling unit is configured to update the motion vector results of the (i-1)th layer. Downsampling is performed in each dimension to obtain the motion vector prediction result of the i-th layer.

[0030] The image downsampling unit is configured to downsample the current image C of the (i-1)th layer. i-1 and reference image P i-1 Perform downsampling to obtain the current image C of the i-th layer. i and reference image P i ;

[0031] The second computing unit is configured to perform calculations according to the C. i and P i The dense optical flow method is used to calculate the motion vector of each point in the current layer of the pyramid.

[0032] The downsampled Kalman filter unit is configured to build a downsampled Kalman filter model using the motion vector. The transformed observations affect the prediction results of the motion vector. Perform an update to obtain the updated motion vector of the i-th layer after filtering.

[0033] The third computational unit is configured to compute the current image C at the top layer. n-1 For each pixel, update the result using the top-level motion vector. The reference image P it points to n-1 A block matching algorithm, centered on the location, performs a full search in a local region, and the resulting motion vector from the block matching search is used as the secondary update result of the top-level motion vector.

[0034] The motion vector upsampling unit is configured to initialize j = n-2 and perform a second update on the motion vector of the (j+1)th layer. Upsample each dimension to obtain the preliminary prediction result of the motion vector of the j-th layer.

[0035] The fourth computational unit is configured to compute the current image C at the j-th layer. j For each pixel, the preliminary prediction result of the motion vector of the j-th layer is obtained. The reference image P it points to j The block matching algorithm, centered on the j-th position, performs a full search in a local region to obtain the motion vector of the block matching search as the final prediction result of the motion vector of the j-th layer.

[0036] The upsampling Kalman filter unit is configured to build an upsampling Kalman filter model using the update results of the motion vectors of the j-th layer. The final prediction result of the transformed observations on the motion vector of the j-th layer. Perform an update to obtain the secondary update result of the motion vector of the j-th layer after filtering.

[0037] Optionally, for the motion vector downsampling unit, if the current layer is the bottom layer of the pyramid, the input motion vector is the output of the first calculation unit; otherwise, the input motion vector is the output of the downsampling Kalman filter unit.

[0038] Optionally, for the motion vector upsampling unit, if the current layer is the top layer of the pyramid, the input motion vector is the output of the third calculation unit; otherwise, the input motion vector is the output of the upsampling Kalman filter unit.

[0039] Based on the same technical concept, the present invention also provides a readable storage medium having a computer program stored thereon, which, when executed, can implement the motion vector estimation method described above.

[0040] In the motion vector estimation method, system, and readable storage medium provided by this invention, a multi-layered pyramid structure is utilized to calculate motion vectors from different layers. Dense optical flow is used for downsampling motion vector calculation, and Kalman filtering is applied layer by layer from the bottom to the top for iterative updates. During upsampling, a local block matching algorithm is used to update the motion vector based on the region pointed to by the motion vector update result of the upsampling layer. The motion vector is then corrected using the filtered motion vector obtained during the process from the bottom to the top. This iterative update using Kalman filtering from the top to the bottom achieves a clean and accurate motion vector. This method combines the advantages of two motion vector calculation algorithms and compensates for their respective disadvantages. Through iterative processing from the bottom to the top and back to the bottom within the complete pyramid structure, clean and relatively accurate motion vectors can be quickly obtained even when processing videos with poor signal-to-noise ratios. Attached Figure Description

[0041] Those skilled in the art will understand that the accompanying drawings are provided to better understand the invention and do not constitute any limitation on the scope of the invention. Wherein:

[0042] Figure 1 A flowchart of a motion vector estimation method provided in an embodiment of the present invention;

[0043] Figure 2 This is a system block diagram of a motion vector estimation system provided in an embodiment of the present invention.

[0044] in:

[0045] 100 - Data input unit; 101 - First calculation unit; 102 - Motion vector downsampling unit; 103 - Image downsampling unit; 104 - Second calculation unit; 105 - Downsampling Kalman filter unit; 106 - Third calculation unit; 107 - Motion vector upsampling unit; 108 - Fourth calculation unit; 109 - Upsampling Kalman filter unit. Detailed Implementation

[0046] To make the objectives, advantages, and features of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that the drawings are in a very simplified form and use non-precise proportions, and are only used to facilitate and clearly illustrate the purpose of the embodiments of this invention. Please refer to the accompanying drawings to make the objectives, features, and advantages of this invention more apparent and understandable. It should be understood that the structures, proportions, sizes, etc., depicted in the accompanying drawings are only used to complement the content disclosed in the specification, for those skilled in the art to understand and read, and are not intended to limit the implementation conditions of this invention. Any modifications to the structure, changes in proportions, or adjustments to the size, if they are the same as or similar to the effects and objectives achieved by this invention, should still fall within the scope of the technical content disclosed in this invention.

[0047] As used herein, the singular forms “a,” “an,” and “the” include plural objects unless otherwise expressly stated. As used herein, the term “or” is generally used to mean “and / or” unless otherwise expressly stated. As used herein, the term “a number” is generally used to mean “at least one” unless otherwise expressly stated. As used herein, the term “at least two” is generally used to mean “two or more” unless otherwise expressly stated.

[0048] Please refer to Figure 1 This invention provides a motion vector estimation method, comprising the following steps:

[0049] S1. Obtain the current image C0 to be processed and the reference image P0;

[0050] S2. Calculate the update results of the motion vector of each point in the current layer of the pyramid using the dense optical flow method. And initialize i = 1;

[0051] S3. Update results of motion vectors for the (i-1)th layer Downsampling is performed in each dimension to obtain the motion vector prediction result of the i-th layer.

[0052] S4. For the current image C in the (i-1)th layer i-1 and reference image P i-1 Perform downsampling to obtain the current image C of the i-th layer. i and reference image P i ;

[0053] S5, according to the aforementioned C i and P i, calculate the motion vector of each point in the current layer of the pyramid using the dense optical flow method

[0054] S6. Establish a downsampled Kalman filter model and use the motion vector The converted observation quantity updates the prediction result of the motion vector to obtain the updated result of the motion vector of the i-th layer after filtering

[0055] S7. Set the total number of layers of the pyramid to n, let i = i + 1. If i < n, repeat steps S3 - S6. If i = n, execute step S8 to obtain the updated result of the motion vector of the top layer of the pyramid after filtering

[0056] S8. For each pixel point of the current image C of the top layer n-1 using the updated result of the motion vector of the top layer to point to the reference image P n-1 perform a full-search block matching algorithm in the local area with the position as the center, and obtain the motion vector of the block matching search as the secondary updated result of the motion vector of the top layer

[0057] S9. Initialize j = n - 2, and upsample each dimension of the secondary updated result of the motion vector of the (j + 1)-th layer to obtain the preliminary prediction result of the motion vector of the j-th layer

[0058] S10. For each pixel point of the current image C of the j-th layer j using the preliminary prediction result of the motion vector of the j-th layer to point to the reference image P j perform a full-search block matching algorithm in the local area with the position as the center, and obtain the motion vector of the block matching search as the final prediction result of the motion vector of the j-th layer

[0059] S11. Establish an upsampled Kalman filter model and use the updated result of the motion vector of the j-th layer The converted observation quantity updates the final prediction result of the motion vector of the j-th layer to obtain the secondary updated result of the motion vector of the j-th layer after filtering

[0060] S12. Let j = j - 1. If j ≥ 0, repeat steps S9 - S11. If j < 0, is the final result of the motion vector estimation.

[0061] ​In the motion vector estimation method provided in this embodiment of the invention, during downsampling, the advantages of optical flow algorithm—low computational cost and small receptive field—are utilized to quickly calculate the motion vector results for each layer on a large scale. Simultaneously, Kalman filtering at different levels and mean filtering at the same level continuously improve the signal-to-noise ratio of the motion vector, thus ensuring a relatively clean and accurate motion vector at the top of the pyramid. During upsampling, the accuracy of block matching and the motion vector reference value provided by the previous layer are used as a starting point for a local full search. The computational cost is less than that of traditional full-scale fast block matching algorithms. Furthermore, upsampling Kalman filtering is continuously performed using the optical flow motion vectors from the same layer after downsampling Kalman filtering to correct for errors, reducing the matching error propagated layer by layer in the downsampling block matching process. Finally, an accurate and clean motion vector is obtained at the bottom layer.

[0062] This invention utilizes a multi-layered pyramid structure to calculate motion vectors from different layers. Downsampling motion vector calculation employs dense optical flow, iteratively updating layer by layer from the bottom to the top using Kalman filtering. During upsampling, a local block matching algorithm is used to update the motion vector based on the region indicated by the motion vector update results of the upsampling layer. Correction is then performed using the filtered motion vectors obtained during the bottom-to-top process, iteratively updating layer by layer from the top to the bottom using Kalman filtering to achieve clean and accurate motion vectors. This combines the advantages of two motion vector calculation algorithms while compensating for their respective disadvantages. Through iterative processing from bottom to top and back to bottom within the complete pyramid structure, it can quickly obtain clean and relatively accurate motion vectors even when processing videos with very poor signal-to-noise ratios.

[0063] In this embodiment, the motion vector is a two-dimensional vector. Therefore, the Kalman filter model between each two layers is a two-dimensional equation. Since the motion vector is calculated by the algorithm, it cannot be used as an observation. Therefore, this application uses the optical flow equation to use the frame difference as an observation and establishes a two-dimensional to one-dimensional observation matrix. This allows the establishment of a Kalman filter model that conforms to the theory, making the Kalman filtering of the motion vector more effective.

[0064] First, step S1 is executed to obtain the current image C0 to be processed and the reference image P0. Preferably, the reference image is the previous frame of the current image after spatiotemporal domain noise reduction processing.

[0065] Next, step S2 is executed, using the dense optical flow method to calculate the update result of the motion vector of each point in the current layer (layer 0) of the pyramid. And initialize i = 1.

[0066] Then, step S3 is executed to update the motion vector results for the (i-1)th layer. Downsampling is performed in each dimension to obtain the motion vector prediction results of the i-th layer, where the height and width are each half the original dimensions.

[0067] Then execute step S4 to process the current image C of layer i-1. i-1 and reference image P i-1 Perform downsampling to obtain the current image C of the i-th layer, whose height and width are each half the original image size. i and reference image P i .

[0068] In this embodiment, the downsampling method used in steps S3 and S4 is 2x2 mean downsampling. The image is divided into 2x2 blocks, and for each sub-block, the mean of the four elements in that block is calculated; this is the downsampled value for that block. The formula is as follows:

[0069] C i (h i ,w j )=(C i-1 (h i *2,w j *2)+C i-1 (h i *2,w j *2+1)+

[0070] C i-1 (h i *2+1,w j *2)+C i-1 (h i *2+1,w j *2+1)) / 4

[0071] P i (h i ,w j )=(P i-1 (h i *2,w j *2)+P i-1 (h i *2,w j *2+1)+

[0072] P i-1 (h i *2+1,w j *2)+P i-1 (h i *2+1,w j *2+1)) / 4

[0073]

[0074] Where i = 1, ..., n-1, (h i ,w j ) represents the h-th level of the i-th layer. i Line w j Column position, C i Let P be the current image of the i-th layer of the pyramid. i This is a reference image of the j-th level of the pyramid. The output of step S3 is the motion vector prediction result. The updated result of the motion vector of the (i-1)th layer after being filtered by the downsampled Kalman filter model. All motion vectors are two-dimensional vectors.

[0075] Then proceed to step S5, according to the C. i and P i The dense optical flow method is used to calculate the motion vector of each point in the current layer of the pyramid.

[0076] In steps S2 and S5, the dense optical flow method is used to calculate the motion vector of the current i-th layer. and i = 1, ..., n-1. The optical flow method assumes that the motion vectors in the local region are the same. Let the spatial position of the point to be processed be (h... x ,w y The local region centered on the point to be processed is defined as [flow_size, flow_size]. `flow_size` is set based on hardware implementation costs and desired effects; in this implementation example, `flow_size` is set to 5. First, the horizontal gradient, vertical gradient, and pixel difference of each pixel in the local region are calculated. The value range is [h...]. x -2,h x +2] points The range of values ​​is [w y -2,w y +2] points

[0077] Pixel difference is calculated using the following formula.

[0078]

[0079] The gradient formula is calculated using the following operators:

[0080]

[0081]

[0082] The formulas for the horizontal and vertical gradients of the current image are:

[0083]

[0084]

[0085] According to the optical flow formula, we have:

[0086]

[0087] The 5x5 local region has 25 points, so the following set of optical flow equations can be listed.

[0088]

[0089] Where p i p j The values ​​range from -2 to 2, mv h ,mv w For each dimension of the motion vector to be determined.

[0090] The solution can be obtained using the least squares method, as shown in the following formula:

[0091]

[0092]

[0093]

[0094] Where, if dm i If mv_w = 0, then mv_h = 0. Otherwise, calculate using the following formula:

[0095]

[0096]

[0097] Next, step S6 is executed to establish a downsampled Kalman filter model, utilizing motion vectors. Transformed observations on motion vector prediction results Perform an update to obtain the updated motion vector of the i-th layer after filtering. Specifically, firstly, based on the result of step S3 A prediction model is established, and then a measurement matrix is ​​designed based on the optical flow model. Next, the Kalman gain is calculated, and the results from step S5 are used to determine the final result. Transformed observation pairs Perform an update. Obtain the updated motion vector of the i-th layer after filtering.

[0098] In this embodiment, a downsampled Kalman filter model is established in step S6. Let the current layer be the i-th layer of the pyramid, i = 1, ..., n-1. All motion vectors are uniformly 2x1 vectors. The downsampled Kalman filter model is as follows:

[0099] a) The state prediction equation is:

[0100]

[0101] The above process is the mean downsampling process in step S3.

[0102] b) Variance prediction equation, which can be easily derived from the state prediction equation as follows:

[0103]

[0104] Among them, Q(h) x ,w y Let Q(h) be the process noise covariance of 2x2. Assuming the downsampling process is accurate, the variance of the disturbance term is small. Typically, we set Q(h) to be... x ,w y ) can be a zero vector. Depend on This is derived in step e below. And from It is derived from, among which Let be the variance matrix of the motion vectors after Kalman filtering. Therefore, according to the recursive formula, we only need to initialize the variance when i=0 to derive it sequentially. The dimension is 2x2. Let's assume the two dimensions of the direction of motion are independent. It is a diagonal matrix, that is:

[0105]

[0106] in, Let (h) be the variances of the values ​​in the two dimensions of the original motion vector, respectively. Assume the current point is (h... x ,w y If the vertical and horizontal gradients of the current point are respectively... and Based on the optical flow equation, the variance optical flow equation can be derived as follows:

[0107]

[0108] Where, varDif 0 (h x ,w y The value (Dif) represents the difference between a pixel in the current image and a pixel in the reference image at the same location. 0 (h x ,wy The variance of is the sum of the variance of the current image pixel and the variance of the reference image pixel at the same position.

[0109] Right now

[0110] varDif 0 (h x ,w y ) = varC i (h x ,w y )+varP i (h x ,w y )

[0111] By assuming that the motion vectors are the same in the local region, the optical flow variance equation for each point in the local region can be similarly listed, and then solved using the least squares method. and By substituting the values ​​into the variance prediction equation, the variance matrix for each point can be obtained. In the subsequent step e, it can be derived from... Find Therefore, the i-th layer can be obtained after multiple iterations.

[0112] c) The Kalman gain equation is:

[0113]

[0114] Among them, GC i (h x ,w y The vector is 1x2, consisting of the horizontal and vertical gradients of the current point, i.e.:

[0115]

[0116] In the downsampling Kalman filter model of this embodiment, GC i (h x ,w y ) is used as the observation matrix H.

[0117] R(h x ,w y The measurement noise covariance is 1x1, representing the error between the measured data and the actual data, i.e., the vardif(h) mentioned in step b. x ,w y The final step c outputs a 2x1 Kalman gain K.

[0118] d) The state update equation is:

[0119]

[0120] e) The variance update equation is as follows:

[0121]

[0122] Then, step S7 is executed. Set the total number of pyramid layers to n, let i = i + 1. If i < n, repeat steps S3 - S6. If i = n, execute step S8 to obtain the updated result of the motion vector at the top layer of the filtered pyramid. In this embodiment, according to the downsampled Kalman filter model, Kalman filtering is performed on the motion vectors of the downsampled frame, and the result is continuously used as the input for the next layer of the frame. The Kalman filter model is iteratively applied until the motion vector result at the top layer is output.

[0123] Then, step S8 is executed. For the current image C at the top layer n-1 For each pixel point, using the updated result of the motion vector at the top layer The position of the reference image P n-1 As the center, perform a full - search block - matching algorithm in the local area to obtain the motion vector of the block - matching search as the secondary updated result of the motion vector at the top layer. In this embodiment, according to the updated result of the motion vector at the top layer of the pyramid output in step S7 For the current image C at the top layer n-1 For each pixel point, the position of the pixel point of the reference image P at the top layer pointed to by its motion vector update result n-1 As the center, perform a full - search block - matching algorithm within the local window of [s_size, s_size]. The size of the matching window is [m_size, m_size], and output the secondary updated result of the motion vector corresponding to the best result of the block - matching. Among them, the values of s_size and m_size depend on the cost of hardware implementation. In this embodiment, s_size is taken as 5, m_size is taken as 5, and the similarity calculation method of the block - matching adopts the Manhattan distance calculation.

[0124] Then, step S9 is executed. Initialize j = n - 2, and upsample each dimension of the secondary updated result of the motion vector at the (j + 1) - th layer To obtain the preliminary prediction result of the motion vector at the j - th layer

[0125] Preferably, bilinear interpolation sampling is performed on each dimension of the secondary updated result of the motion vector at the (j + 1) - th layer To obtain the preliminary prediction result of the motion vector at the j - th layer with the height and width sizes each being twice the height and width sizes of the original frame.

[0126] Then, step S10 is executed for the current image C of the j-th layer. j For each pixel, the preliminary prediction result of the motion vector of the j-th layer is obtained. The reference image P it points to j The block matching algorithm, centered on the j-th position, performs a full search in a local region to obtain the motion vector of the block matching search as the final prediction result of the motion vector of the j-th layer. In this embodiment, a block matching algorithm with full search is performed within a local window of [s_size, s_size], and the size of the matching window is [m_size, m_size].

[0127] Next, step S11 is executed to establish an upsampled Kalman filter model, using the update results of the motion vector of the j-th layer. The final prediction result of the transformed observations on the motion vector of the j-th layer. Perform an update to obtain the secondary update result of the motion vector of the j-th layer after filtering. Specifically, based on the results of steps S6 and S10, an upsampled Kalman filter model for the motion vector is established. First, based on the results of step S10... A prediction model is established, and then a measurement matrix is ​​designed based on the optical flow model. Next, the Kalman gain is calculated, and the results from step S6 are used to determine the final measurement. The final prediction result of the transformed observations on the motion vector of the j-th layer. Perform an update. Obtain the secondary update result of the motion vector of the j-th layer after filtering.

[0128] In this embodiment, the upsampling Kalman filter model in step S11 adopts a similar construction method to the downsampling Kalman filter model in step S6, as follows:

[0129] a) The state prediction equation is changed from the mean downsampling prediction equation to the bilinear interpolation upsampling prediction equation, i.e.:

[0130]

[0131]

[0132] Where bilinear() is a bilinear interpolation algorithm, which upsamples by 2 times in both the horizontal and vertical directions, i.e., the process in step S9.

[0133] block_mc() is used as The optimal motion vector is found locally using a block matching algorithm with the pointed position as the center, which is the process of step S10.

[0134] b) In the variance prediction equation, the variance of the top layer... The variance passed from the top layer of the downsampled Kalman filter model approximate.

[0135] During bilinear interpolation, the motion vector of each point on the upsampled frame can be derived from the weighting coefficient of the four corresponding motion vectors on the current frame, denoted as w1. j w2 j w3 j w4 j The variances of these four motion vectors are denoted as follows:

[0136] The variance of the motion vector upsampled by bilinear interpolation The formula for each dimension is as follows:

[0137]

[0138]

[0139] To simplify calculations, it is assumed that the variance remains constant during block matching. Therefore, the variance prediction equation for upsampling Kalman filtering is the formula described above.

[0140] c) The Kalman gain equation is the same as that in the downsampling Kalman filter model, that is:

[0141]

[0142] d) The state update equation changes the motion vectors of the input and output, that is:

[0143]

[0144] e) The formula for the variance update equation is the same as that in the downsampling Kalman filter model, that is:

[0145]

[0146] Finally, execute step S12, letting j = j-1. If j ≥ 0, then repeat steps S9-S11; if j < 0, then... The final result of motion vector estimation is the motion vector estimation result of the current image to be processed.

[0147] Based on this, embodiments of the present invention also provide a motion vector estimation system for implementing the motion vector estimation method described above, such as... Figure 2 As shown, it includes:

[0148] The data input unit 100 is configured to input the current image C0 to be processed and the reference image P0;

[0149] The first computing unit 101 is configured to calculate the update result of the motion vector of each point in the current layer of the pyramid using the dense optical flow method. And initialize i = 1;

[0150] Motion vector downsampling unit 102 is configured to update the motion vector results of layer (i-1). Downsampling is performed in each dimension to obtain the motion vector prediction result of the i-th layer.

[0151] Image downsampling unit 103 is configured to downsampling the current image C of layer (i-1). i-1 and reference image P i-1 Perform downsampling to obtain the current image C of the i-th layer. i and reference image P i ;

[0152] The second computing unit 104 is configured to calculate according to the C i and P i The dense optical flow method is used to calculate the motion vector of each point in the current layer of the pyramid.

[0153] The downsampled Kalman filter unit 105 is configured to build a downsampled Kalman filter model using motion vectors. Transformed observations on motion vector prediction results Perform an update to obtain the updated motion vector of the i-th layer after filtering.

[0154] The third computing unit 106 is configured to compute the current image C at the top layer. n-1 For each pixel, update the result using the top-level motion vector. The reference image P it points to n-1 A block matching algorithm, centered on the location, performs a full search in a local region, and the resulting motion vector from the block matching search is used as the secondary update result of the top-level motion vector.

[0155] Motion vector upsampling unit 107 is configured to initialize j = n-2 and perform a second update of the motion vector results for the (j+1)th layer. Upsample each dimension to obtain the preliminary prediction result of the motion vector of the j-th layer.

[0156] The fourth computing unit 108 is configured to compute the current image C of the j-th layer. j For each pixel, the preliminary prediction result of the motion vector of the j-th layer is obtained. The reference image P it points toj The block matching algorithm, centered on the j-th position, performs a full search in a local region to obtain the motion vector of the block matching search as the final prediction result of the motion vector of the j-th layer.

[0157] The upsampling Kalman filter unit 109 is configured to build a downsampling Kalman filter model using the update results of the motion vector of the j-th layer. The final prediction result of the transformed observations on the motion vector of the j-th layer. Perform an update to obtain the secondary update result of the motion vector of the j-th layer after filtering.

[0158] In this embodiment, the first calculation unit 101 is used to calculate the horizontal gradient, vertical gradient and pixel difference of each position in the local region of the point to be processed and the same position of the reference image, and to establish a dense optical flow equation system and use the least squares method to solve the initial value of the motion vector of the original surface.

[0159] In this embodiment, the motion vector downsampling unit 102 uses a mean downsampling method to divide the input motion vector into different 2x2 blocks. The mean of each dimension of the four motion vectors in each 2x2 block is taken to obtain the motion vector of its corresponding downsampled area. If the current layer is the bottom layer of the pyramid, the input motion vector is the output of the first calculation unit 101; otherwise, the input motion vector is the output of the downsampling Kalman filter unit 105.

[0160] In this embodiment, the image downsampling unit 103 employs a mean downsampling method, dividing the input current image and reference image of the current layer into different 2x2 blocks. The average of the four pixel values ​​of each 2x2 block is taken to obtain the pixel values ​​of the downsampling area corresponding to the current image and the pixel values ​​of the downsampling area corresponding to the reference image. If the current layer is the bottom layer of the pyramid, the input current image and reference image are the current image C0 and the reference image P0; otherwise, they are the results of its most recent output.

[0161] In this embodiment, the second calculation unit 104 is used to calculate the horizontal gradient, vertical gradient, and pixel difference of each position in the local area of ​​the point to be processed on the current sampling area, and establish a dense optical flow equation system, and use the least squares method to solve the motion vector of the current sampling area.

[0162] In this embodiment, the downsampling Kalman filter unit 105 uses the motion vector output by the second calculation unit 104 and the motion vector output by the motion vector downsampling unit 102 to establish a downsampling Kalman filter model, and outputs the result of the motion vector of the current layer after Kalman filtering. If the current layer is the top layer of the pyramid, the result is output to the third calculation unit 106; otherwise, the result is output to the motion vector downsampling unit 102.

[0163] In this embodiment, the third calculation unit 106 is used to update the block matching result of the motion vector of the top layer of the pyramid based on the position of the top layer reference image pointed to by the motion vector of the top layer of the pyramid output by the downsampling Kalman filter unit 105, and using a full search block matching algorithm in a local area with that position as the center.

[0164] In this embodiment, the motion vector upsampling unit 107 is used to perform an upsampling operation on the input motion vector. The upsampling method is to perform bilinear interpolation on each dimension of the motion vector to obtain a motion vector result that is upsampled by a factor of 2 in both the horizontal and vertical directions. If the current layer is the top layer of the pyramid, the input motion vector is the output of the third calculation unit 106; otherwise, the input motion vector is the output of the upsampling Kalman filter unit 109.

[0165] In this embodiment, the fourth calculation unit 108 is used to obtain the motion vector of the current layer's block matching method by using a full-search block matching algorithm in a local area, based on the position of the reference image of the current layer pointed to by the motion vector output by the motion vector upsampling unit 107 and taking that position as the center.

[0166] In this embodiment, the upsampling Kalman filter unit 109 uses the motion vector output by the fourth calculation unit 108 and the motion vector output by the downsampling Kalman filter unit 105 to establish an upsampling Kalman filter model and output the result of the motion vector of the current layer after Kalman filtering. If the current layer is the bottom layer of the pyramid, the final motion vector result of the current layer is output; otherwise, it is output to the motion vector upsampling unit 107.

[0167] Based on the same inventive concept, embodiments of the present invention also propose a readable storage medium on which a computer program is stored, which, when executed, can implement the motion vector estimation method described above.

[0168] A readable storage medium can be a tangible device capable of holding and storing instructions for use by an instruction execution device, such as, but not limited to, electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof. The computer programs described herein can be downloaded from the readable storage medium to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. Networks can include copper transmission cables, fiber optic transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. Each computing / processing device's network adapter card or network interface receives and forwards a computer program from the network for storage on a readable storage medium within the respective computing / processing device. The computer program used to perform the operations of this invention can be execution instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as "C" or similar languages. The computer program can execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuits, such as programmable logic circuits, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), are personalized by utilizing state information from a computer program. These electronic circuits can execute computer-readable program instructions, thereby realizing various aspects of the present invention.

[0169] Various aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, systems, and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by a computer program. These computer programs can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams. These computer programs can also be stored in a readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the readable storage medium storing the computer program comprises an article of manufacture including instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams.

[0170] A computer program may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the computer program executing on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0171] In summary, this invention provides a motion vector estimation method, system, and readable storage medium. It improves upon the block matching algorithm of a pyramid structure by establishing prediction and update equations for motion vectors between adjacent layers. This allows for Kalman filtering of the motion vectors layer by layer to correct them, addressing the problem that erroneous motion vectors calculated by traditional pyramid block matching are propagated layer by layer. Simultaneously, an optical flow algorithm based on a downsampled pyramid structure is additionally established. While still employing Kalman filtering, it updates the motion vectors layer by layer from the bottom to the top. At the top layer, using the updated motion vector as the center, it locally updates the motion vector using a full-search block matching method. This solves the problem of poor accuracy in calculating the top-level motion vector in traditional pyramid block matching, resulting in a cleaner and more accurate final output motion vector.

[0172] The above description is merely a description of preferred embodiments of the present invention and is not intended to limit the scope of the invention in any way. Any changes or modifications made by those skilled in the art based on the above disclosure are within the protection scope of the present invention. Obviously, those skilled in the art can make various modifications and variations to the present invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the present invention and its equivalents, the present invention also intends to include these modifications and variations.

Claims

1. A method for estimating motion vectors, characterized in that, include: S1. Obtain the current image C0 to be processed and the reference image P0; S2. Calculate the update results of the motion vector of each point in the current layer of the pyramid using the dense optical flow method. Initialize i = 1; S3. Update results of motion vectors for the (i-1)th layer Downsampling is performed in each dimension to obtain the motion vector prediction result of the i-th layer. S4. For the current image C in the (i-1)th layer... i-1 and reference image P i-1 Perform downsampling to obtain the current image C of the i-th layer. i and reference image Pi; S5, according to the aforementioned C i and P i The dense optical flow method is used to calculate the motion vector of each point in the current layer of the pyramid. S6. Establish a downsampled Kalman filter model, using the motion vector. The transformed observations affect the prediction results of the motion vector. Perform an update to obtain the updated motion vector of the i-th layer after filtering. S7. Set the total number of pyramid layers as n, let i = i + 1. If i < n, repeat steps S3 - S6. If i = n, execute step S8 to obtain the updated result of the motion vectors at the top layer of the filtered pyramid S8. For the current top-level image C n-1 For each pixel, update the result using the top-level motion vector. The reference image P it points to n-1 A block matching algorithm, centered on the location, performs a full search in a local region, and the resulting motion vector from the block matching search is used as the secondary update result of the top-level motion vector. S9. Initialize j = n-2, and update the motion vector of the (j+1)th layer twice. Upsample each dimension to obtain the preliminary prediction result of the motion vector of the j-th layer. S10, For the current image C of the j-th layer j For each pixel, the preliminary prediction result of the motion vector of the j-th layer is obtained. The reference image P it points to j The block matching algorithm, centered on the j-th position, performs a full search in a local region to obtain the motion vector of the block matching search as the final prediction result of the motion vector of the j-th layer. S11. Establish an upsampling Kalman filter model and utilize the update results of the motion vectors in the j-th layer. The final prediction result of the transformed observations on the motion vector of the j-th layer. Perform an update to obtain the secondary update result of the motion vector of the j-th layer after filtering. S12. Let j = j-1. If j ≥ 0, then repeat steps S9-S11. If j < 0, then... This is the final result of motion vector estimation.

2. The motion vector estimation method according to claim 1, characterized in that, The reference image is the previous frame of the current image after spatiotemporal noise reduction processing.

3. The motion vector estimation method according to claim 1, characterized in that, In steps S3 and S4, the downsampling method used is 2x2 mean downsampling.

4. The motion vector estimation method according to claim 1, characterized in that, In step S9, the upsampling method used is bilinear interpolation upsampling.

5. The motion vector estimation method according to claim 1, characterized in that, In step S6, when establishing the downsampled Kalman filter model of the motion vector, the result of step S3 is first used as a basis. Design a prediction model, then design a measurement matrix based on the optical flow algorithm, and then calculate the Kalman gain. The results from step S5 are then used to... The transformed observations for the Update.

6. The motion vector estimation method according to claim 1, characterized in that, In step S11, when establishing the upsampled Kalman filter model of the motion vector, the result of step S10 is first used as a basis. A prediction model is established, a measurement matrix is ​​designed based on the optical flow algorithm, and then the Kalman gain is calculated. The results of step S6 are then used to determine the final measurement model. The transformed observations update the motion vector.

7. A motion vector estimation system for implementing the motion vector estimation method according to any one of claims 1-6, characterized in that, include: The data input unit is configured to input the current image C0 to be processed and the reference image P0; The first computational unit is configured to use dense optical flow to calculate the update results of the motion vector of each point in the current layer of the pyramid. And initialize i = 1; The motion vector downsampling unit is configured to update the motion vector results of the (i-1)th layer. Downsampling is performed in each dimension to obtain the motion vector prediction result of the i-th layer. The image downsampling unit is configured to downsample the current image C of the (i-1)th layer. i-1 and reference image P i-1 Perform downsampling to obtain the current image C of the i-th layer. i and reference image Pi; The second computing unit is configured to perform calculations according to the C. i Pi, using dense optical flow, calculates the motion vector of each point in the current layer of the pyramid. The downsampled Kalman filter unit is configured to build a downsampled Kalman filter model using the motion vector. The transformed observations affect the prediction results of the motion vector. Perform an update to obtain the updated motion vector of the i-th layer after filtering. The third computational unit is configured to compute the current image C at the top layer. n-1 For each pixel, update the result using the top-level motion vector. The reference image P it points to n-1 A block matching algorithm, centered on the location, performs a full search in a local region, and the resulting motion vector from the block matching search is used as the secondary update result of the top-level motion vector. The motion vector upsampling unit is configured to initialize j = n-2 and perform a second update on the motion vector of the (j+1)th layer. Upsample each dimension to obtain the preliminary prediction result of the motion vector of the j-th layer. The fourth computational unit is configured to compute the current image C at the j-th layer. j For each pixel, the preliminary prediction result of the motion vector of the j-th layer is obtained. The reference image P it points to j The block matching algorithm, centered on the j-th position, performs a full search in a local region to obtain the motion vector of the block matching search as the final prediction result of the motion vector of the j-th layer. The upsampling Kalman filter unit is configured to build an upsampling Kalman filter model using the update results of the motion vectors of the j-th layer. The final prediction result of the transformed observations on the motion vector of the j-th layer. Perform an update to obtain the secondary update result of the motion vector of the j-th layer after filtering.

8. The motion vector estimation system according to claim 7, characterized in that, For the motion vector downsampling unit, if the current layer is the bottom layer of the pyramid, the input motion vector is the output of the first calculation unit; otherwise, the input motion vector is the output of the downsampling Kalman filter unit.

9. The motion vector estimation system according to claim 7, characterized in that, For the motion vector upsampling unit, if the current layer is the top layer of the pyramid, the input motion vector is the output of the third calculation unit; otherwise, the input motion vector is the output of the upsampling Kalman filter unit.

10. A readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed, it can implement the motion vector estimation method according to any one of claims 1-6.

Images