Motion vector estimation method and system and readable storage medium

Abstract

The invention relates to a motion vector estimation method and system and a readable storage medium, and the method comprises the steps: calculating a motion vector from different layers through employing a multi-layer structure of a pyramid, employing a dense optical flow for the calculation of a down-sampled motion vector, and carrying out the layer-by-layer updating and continuous iteration through employing Kalman filtering from the bottommost layer to the topmost layer, during up-sampling, a local block matching algorithm is performed according to a region pointed by a motion vector updating result of an up-sampling layer to update a motion vector, then correction is performed through a filtered motion vector obtained in a process from the bottommost layer to the topmost layer, and layer-by-layer updating and continuous iteration are performed by using Kalman filtering from the topmost layer to the bottommost layer. And the effect of obtaining a clean and accurate motion vector is achieved. Therefore, the advantages of the two algorithms for calculating the motion vector are combined, the defects of the two algorithms are mutually compensated, and through iteration from the bottommost layer to the topmost layer and then to the bottommost layer in the complete structure of the pyramid, the clean and accurately discriminated motion vector can be quickly obtained when a video with a very poor signal-to-noise ratio is processed finally.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a motion vector estimation method, system and readable storage medium. Background Art

[0002] Motion vector estimation is widely used in image encoding and decoding algorithms and noise reduction algorithms. Its core idea is to solve the reference position of each pixel or pixel block of the current image in the previous frame through a correlation algorithm. The direction of the difference between the reference position and the position of the pixel or pixel block of the current image is the direction of the motion vector; the size of the difference is the size of the motion vector. The ultimate goal of motion vector estimation is to find the position of the pixel or pixel block in the previous frame of the current image for the pixel point or pixel block to be processed, and use the pixel point or pixel block at this position as a reference to predict or reduce noise for the pixel point or pixel block to be processed in the current image. For example, in encoding algorithms such as h264 and h265, a reference pixel block is found as the prediction result of the current block to be processed, and the residual between the prediction result and the actual result is calculated, and the residual is encoded. In the end, only the encoding result of the motion vector and the encoding result of the residual need to be stored, which can greatly improve the compression rate while ensuring its quality. In the noise reduction algorithm, the traditional time domain filtering will control the noise reduction intensity according to whether the current position is moving. If it is still, the average value of the pixels at the same position as the reference image is taken to achieve the purpose of improving the signal-to-noise ratio. If it is moving, no noise reduction is performed. However, if the motion vector of the object is known, the position of the point to be processed or the block to be processed in the current image in the reference frame is found in the reference image, and then the reference pixel block or reference point at that position is moved to the position of the block to be processed. After traversing the entire image, a deformed reference image can be obtained. The pixels or pixel blocks at the same position of the deformed reference image and the current image are theoretically the same, so the current image can be subjected to time domain filtering based on the deformed reference image. Compared with the traditional time domain noise reduction method, the use of a deformed reference image can achieve the effect of simultaneously reducing noise for both still and moving objects.

[0003] There are two mainstream methods for motion vector estimation: motion vector estimation algorithm based on block matching and motion vector estimation method based on optical flow.

[0004] The premise of block matching is that the movement of most moving objects in each frame is basically slow, that is, the position of the previous frame of the current processing point is within the local area of ​​the current position. By limiting the search range, all possible points in the search range are compared with the current point to find the best matching point. Block matching usually limits the movement of the local small area of ​​the object to only translation, without rotation, stretching and other movements. Therefore, the block of the local range with the current processing point as the center and the block of the same size with each reference point in the search range as the center are calculated to find the similarity. The method for calculating the similarity usually uses the relatively simple Manhattan distance. The reference point with the greatest similarity is found by comparison as the best matching point. Although the formula for calculating the similarity algorithm uses the simple Manhattan distance, the search method is full search. If the search range is large, it is necessary to calculate the similarity of a large number of reference blocks at the same time, which is difficult to achieve real-time. Therefore, some fast block matching algorithms are needed. Fast block matching algorithms usually give a certain search strategy, such as square search, cross search, diamond search, etc., and gradually iterate a few times to find a result close to the optimal. In h265, the motion information of the previous frame and the motion information of the processed points around the current point to be processed are also used to infer the predicted value of the motion vector of the current point, and the position pointed to by the predicted value is used as the search center for searching, further improving the search efficiency.

[0005] Optical flow algorithms are usually divided into sparse optical flow and dense optical flow. Sparse optical flow first finds all corner points in the image, calculates the motion vectors of all corner points, and then interpolates the motion vector of each pixel in the image. Dense optical flow directly solves the motion vector of each pixel in the image. The motion vector solution of optical flow is based on the two assumptions that the illumination of the moving object at adjacent moments remains unchanged and the motion vectors of the local area are the same. First, the optical flow equation is established based on the unchanged illumination of the object at adjacent moments, and the optical flow equation group is established based on the same motion vectors of the local area. Then, the horizontal gradient and vertical gradient of all pixels in the local area are solved based on the current image and the reference image, as well as the difference between the pixels of the current image and the pixels of the reference image at the same position. The above three are brought into the optical flow equation group, and the motion vector can be solved using the least squares method.

[0006] However, both the block matching algorithm and the optical flow algorithm do not consider the noise model. If the image has a lot of noise, both methods will find the wrong motion vector, or the motion vector that is right or wrong but overall very chaotic. The pyramid method can effectively improve the signal-to-noise ratio of the image, so there are also improvements using the pyramid method in the block matching algorithm, that is, the image at the top of the pyramid is subjected to the block matching algorithm, and then the obtained motion vector is upsampled. The position pointed to by it in the next layer is used as the center of the motion search of the next layer, and the full search is continued in the local area, and it is continuously iterated until the bottom layer is reached. Although this method can obtain a relatively clean motion vector, its accuracy is difficult to guarantee. First, because the downsampled format loses high-frequency information, the motion vector found is not necessarily accurate; second, because each time the upsampling is performed for motion vector estimation, it is assumed that the motion vector estimation of the previous layer is accurate, and then the local area of ​​the position pointed to is searched. If the motion vector of the previous layer is inaccurate, the subsequent results will be inaccurate. Therefore, it is necessary to make some improvements in the block matching pyramid model to solve the above two problems. Summary of the invention

[0007] The object of the present invention is to provide a motion vector estimation method, system and readable storage medium, which solve the problem that the traditional motion vector estimation algorithm has poor accuracy due to noise interference in a scene with a poor signal-to-noise ratio.

[0008] To achieve the above object, the present invention provides a motion vector estimation method, comprising:

[0009] S1. Get the current image C to be processed 0 And the reference image P 0 ;

[0010] S2. Use the dense optical flow method to calculate the updated result of the motion vector of each point in the current layer of the pyramid Initialize i=1;

[0011] S3, update result of motion vector of i-1th layer Downsample each dimension to obtain the motion vector prediction result of the i-th layer

[0012] S4, for the current image C of the i-1th layer i-1 and the reference image P i-1 Downsample to obtain the current image C of the i-th layer i and the reference image P i ;

[0013] S5. According to the C i and P i , using dense optical flow to calculate the motion vector of each point in the current layer of the pyramid

[0014] S6. Establish a downsampling Kalman filter model, and use the motion vector converted observation quantity to update the motion vector prediction result to obtain the updated result of the motion vector of the i-th layer after filtering

[0015] S7. Set the total number of pyramid layers to n, let i = i + 1. If i < n, repeat steps S3 - S6. If i = n, execute step S8 to obtain the updated result of the motion vector of the top layer of the pyramid after filtering

[0016] S8. For each pixel of the current image C n-1 at the top layer, use the updated result of the motion vector at the top layer to point to the reference image P n-1 at the center position, perform a full search block matching algorithm in the local area to obtain the motion vector of the block matching search as the secondary updated result of the motion vector at the top layer

[0017] S9. Initialize j = n - 2, and upsample each dimension of the secondary updated result of the motion vector of the (j + 1)-th layer to obtain the preliminary prediction result of the motion vector of the j-th layer

[0018] S10. For each pixel of the current image C j at the j-th layer, use the preliminary prediction result of the motion vector at the j-th layer to point to the reference image P j at the center position, perform a full search block matching algorithm in the local area to obtain the motion vector of the block matching search as the final prediction result of the motion vector at the j-th layer

[0019] S11. Establish an upsampling Kalman filter model, and use the updated result of the motion vector of the j-th layer to convert the observation quantity to update the final prediction result of the motion vector of the j-th layer to obtain the secondary updated result of the motion vector of the j-th layer after filtering

[0020] S12. Let j = j - 1. If j ≥ 0, repeat steps S9 - S11. If j < 0, it is the final result of the motion vector estimation.

[0021] Optionally, the reference image is the image obtained by performing spatio-temporal domain noise reduction processing on the previous frame of the current image.

[0022] Optionally, in step S3 and step S4, the downsampling method adopted is 2x2 mean downsampling.

[0023] Optionally, in step S9, the upsampling method adopted is bilinear interpolation upsampling.

[0024] Optionally, in step S6, when establishing the down-sampling Kalman filter model of the motion vector, firstly according to the result of step S3 Design a prediction model, then design a measurement matrix based on the optical flow algorithm, and then calculate the Kalman gain. The transformed observations are to update.

[0025] Optionally, in step S11, when establishing the up-sampling Kalman filter model of the motion vector, firstly according to the result of step S10 Establish a prediction model, then design a measurement matrix based on the optical flow algorithm, and then calculate the Kalman gain. The transformed observations update the motion vectors.

[0026] Based on the same technical concept, the present invention also provides a motion vector estimation system, comprising:

[0027] A data input unit is configured to input a current image C to be processed. 0 And the reference image P 0 ;

[0028] The first calculation unit is configured to calculate the update result of the motion vector of each point in the current layer of the pyramid using the dense optical flow method And initialize i=1;

[0029] The motion vector downsampling unit is configured to update the motion vector of the i-1th layer Downsample each dimension to obtain the motion vector prediction result of the i-th layer

[0030] The image downsampling unit is configured to downsample the current image C of the i-1th layer i-1 and the reference image P i-1 Downsample to obtain the current image C of the i-th layer i and the reference image P i ;

[0031] The second computing unit is configured to calculate the i and P i , using dense optical flow to calculate the motion vector of each point in the current layer of the pyramid

[0032] The down-sampling Kalman filter unit is configured to establish a down-sampling Kalman filter model using the motion vector The transformed observations are the motion vector prediction results Update to obtain the updated result of the motion vector of the i-th layer after filtering

[0033] The third computing unit is configured to calculate the current image C at the top layer. n-1 For each pixel point, the top-level motion vector is used to update the result The reference image P pointed to n-1 The position of is taken as the center, and a block matching algorithm is performed in a local area to perform a full search, and the motion vector of the block matching search is obtained as the secondary update result of the motion vector of the top layer

[0034] The motion vector upsampling unit is configured to initialize j=n-2 and update the motion vector of the j+1th layer twice Upsample each dimension to obtain the preliminary prediction result of the motion vector of the jth layer

[0035] The fourth computing unit is configured to calculate the current image C of the jth layer. j For each pixel point, the motion vector of the jth layer is used to preliminarily predict the result The reference image P pointed to j The position of is taken as the center, and a block matching algorithm is performed in the local area to obtain the motion vector of the block matching search as the final prediction result of the motion vector of the jth layer.

[0036] The upsampling Kalman filter unit is configured to establish an upsampling Kalman filter model using the updated result of the motion vector of the jth layer The final prediction result of the transformed observation on the motion vector of the jth layer Update to obtain the secondary update result of the motion vector of the jth layer after filtering

[0037] Optionally, for the motion vector downsampling unit, if the current layer is the bottom layer of the pyramid, the input motion vector is the output of the first calculation unit; otherwise, the input motion vector is the output of the downsampling Kalman filter unit.

[0038] Optionally, for the motion vector upsampling unit, if the current layer is the topmost layer of the pyramid, the input motion vector is the output of the third calculation unit; otherwise, the input motion vector is the output of the upsampling Kalman filter unit.

[0039] Based on the same technical concept, the present invention further provides a readable storage medium having a computer program stored thereon, and the computer program can implement the motion vector estimation method as described above when executed.

[0040] In a motion vector estimation method, system and readable storage medium provided by the present invention, a multi-layer structure of a pyramid is used to calculate motion vectors from different layers, dense optical flow is used for downsampling motion vector calculation, and Kalman filtering is used to update layer by layer from the bottom layer to the top layer for continuous iteration, and when upsampling, a local block matching algorithm is used to update the motion vector according to the area pointed to by the motion vector update result of the upsampling layer, and then the filtered motion vector obtained in the process from the bottom layer to the top layer is corrected, and Kalman filtering is used to update layer by layer from the top layer to the bottom layer for continuous iteration, so as to achieve the effect of obtaining a clean and accurate motion vector. Thus, the advantages of the two algorithms for calculating motion vectors are combined and their disadvantages are compensated for each other, and through iteration from the bottom layer to the top layer and then to the bottom layer in the complete structure of the pyramid, a clean and relatively accurate motion vector can be quickly obtained when processing a video with a very poor signal-to-noise ratio. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Those skilled in the art will appreciate that the accompanying drawings are provided for a better understanding of the present invention and do not constitute any limitation on the scope of the present invention.

[0042] Figure 1 A flowchart of a motion vector estimation method provided by an embodiment of the present invention;

[0043] Figure 2 A system block diagram of a motion vector estimation system provided by an embodiment of the present invention.

[0044] in:

[0045] 100-data input unit; 101-first calculation unit; 102-motion vector downsampling unit; 103-image downsampling unit; 104-second calculation unit; 105-downsampling Kalman filter unit; 106-third calculation unit; 107-motion vector upsampling unit; 108-fourth calculation unit; 109-upsampling Kalman filter unit. DETAILED DESCRIPTION

[0046] In order to make the purpose, advantages and features of the present invention clearer, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the accompanying drawings are in a very simplified form and use non-precise proportions, which are only used to conveniently and clearly assist in explaining the purpose of the embodiments of the present invention. In order to make the purpose, features and advantages of the present invention more obvious and easy to understand, please refer to the accompanying drawings. It should be noted that the structure, proportion, size, etc. illustrated in the drawings of this specification are only used to match the content disclosed in the specification for people familiar with this technology to understand and read, and are not used to limit the limiting conditions for the implementation of the present invention. Any modification of the structure, change in the proportional relationship or adjustment of the size, under the same or similar conditions as the effects that can be produced by the present invention and the purposes that can be achieved, should still fall within the scope of the technical content disclosed by the present invention.

[0047] As used in the present invention, the singular forms "a", "an", and "the" include plural referents unless the context clearly indicates otherwise. As used in the present invention, the term "or" is generally used in a sense that includes "and / or", unless the context clearly indicates otherwise. As used in the present invention, the term "several" is generally used in a sense that includes "at least one", unless the context clearly indicates otherwise. As used in the present invention, the term "at least two" is generally used in a sense that includes "two or more", unless the context clearly indicates otherwise.

[0048] Please refer to Figure 1 , an embodiment of the present invention provides a motion vector estimation method, comprising the following steps:

[0049] S1. Get the current image C to be processed 0 And the reference image P 0 ;

[0050] S2. Use the dense optical flow method to calculate the updated result of the motion vector of each point in the current layer of the pyramid And initialize i=1;

[0051] S3, update result of motion vector of i-1th layer Downsample each dimension to obtain the motion vector prediction result of the i-th layer

[0052] S4, for the current image C of the i-1th layer i-1 and the reference image P i-1 Downsample to obtain the current image C of the i-th layer i and the reference image P i ;

[0053] S5. According to the C i and Pi , calculate the motion vectors of each point in the current layer of the pyramid using the dense optical flow method

[0054] S6. Establish a downsampling Kalman filter model and use the motion vectors The transformed observation quantities to the predicted results of the motion vectors Perform an update to obtain the updated results of the motion vectors for the i-th layer after filtering

[0055] S7. Set the total number of pyramid layers to n, let i = i + 1. If i < n, repeat steps S3 - S6. If i = n, execute step S8 to obtain the updated results of the motion vectors for the topmost layer of the pyramid after filtering

[0056] S8. For each pixel point in the current image C of the topmost layer n-1 , use the updated results of the motion vectors of the topmost layer The position of the reference image P pointed to n-1 As the center, perform a full-search block matching algorithm in the local area to obtain the motion vector of the block matching search as the secondary updated results of the motion vectors of the topmost layer

[0057] S9. Initialize j = n - 2, upsample each dimension of the secondary updated results of the motion vectors of the (j + 1)-th layer To obtain the preliminary predicted results of the motion vectors of the j-th layer

[0058] S10. For each pixel point in the current image C of the j-th layer j , use the preliminary predicted results of the motion vectors of the j-th layer The position of the reference image P pointed to j As the center, perform a full-search block matching algorithm in the local area to obtain the motion vector of the block matching search as the final predicted results of the motion vectors of the j-th layer

[0059] S11. Establish an upsampling Kalman filter model and use the updated results of the motion vectors of the j-th layer The transformed observation quantities to the final predicted results of the motion vectors of the j-th layer Perform an update to obtain the secondary updated results of the motion vectors of the j-th layer after filtering

[0060] S12. Let j = j - 1. If j ≥ 0, repeat steps S9 - S11. If j < 0, It is the final result of the motion vector estimation

[0061] In the motion vector estimation method provided by the embodiment of the present invention, the advantages of the optical flow algorithm with less calculation amount and small receptive field are used during downsampling, and the motion vector results of each layer are quickly obtained by using the optical flow in a large format. At the same time, the Kalman filters at different levels and the mean filters at the same level are used to continuously improve the signal-to-noise ratio of the motion vector, thereby jointly ensuring that a relatively clean and accurate motion vector is obtained at the top level of the pyramid. When upsampling, the accuracy of block matching and the motion vector reference value provided by the previous layer are used as the starting point for a local full search. The amount of calculation is less than that of the traditional full-frame fast block matching algorithm, and at the same time, the optical flow motion vector of the same layer after the downsampling Kalman filter is continuously used for upsampling Kalman filtering to correct it, reducing the matching error transmitted layer by layer by the downsampling format block matching, and finally obtaining an accurate and clean motion vector at the bottom layer.

[0062] The present invention utilizes the multi-layer structure of a pyramid to calculate motion vectors from different layers, uses dense optical flow to calculate the motion vector of downsampling, and uses Kalman filtering to update layer by layer and continuously iterate from the bottom layer to the top layer, and uses a local block matching algorithm to update the motion vector according to the area pointed to by the motion vector update result of the upsampling layer during upsampling, and then corrects the filtered motion vector obtained in the process from the bottom layer to the top layer, and uses Kalman filtering to update layer by layer and continuously iterate from the top layer to the bottom layer, so as to achieve the effect of obtaining a clean and accurate motion vector. Thus, the advantages of the two algorithms for calculating motion vectors are combined and their disadvantages are compensated for each other, and through the iteration from the bottom layer to the top layer and then to the bottom layer in the complete structure of the pyramid, a clean and relatively accurate motion vector can be quickly obtained when processing a video with a very poor signal-to-noise ratio.

[0063] In this embodiment, the motion vector is a two-dimensional vector, so the Kalman filter model between every two layers is a two-dimensional equation, and the motion vectors are calculated by the algorithm and cannot be used as observation quantities. Therefore, this application uses the optical flow equation and takes the frame difference as the observation quantity to establish a two-dimensional to one-dimensional observation matrix, so that a Kalman filter model that conforms to the theory can be established, and the Kalman filtering of the motion vector is more effective.

[0064] First, execute step S1 to obtain the current image C to be processed. 0 And the reference image P 0 Preferably, the reference image is an image of a previous frame of the current image that has been subjected to spatiotemporal denoising processing.

[0065] Then, step S2 is performed to calculate the updated result of the motion vector of each point in the current layer (layer 0) of the pyramid using the dense optical flow method. And initialize i=1.

[0066] Then, step S3 is executed to update the motion vector of the i-1th layer. Downsample each dimension to obtain the motion vector prediction result of the i-th layer whose height and width are half of the original height and width

[0067] Then execute step S4 to calculate the current image C of the i-1th layer. i-1 and the reference image P i-1 Downsample to obtain the current image C of the i-th layer, whose height and width are half of the original height and width. i and the reference image P i .

[0068] In this embodiment, in step S3 and step S4, the downsampling method used is 2x2 mean downsampling, by dividing the image into 2x2 blocks, and for each sub-block, the mean of the four elements in the block is calculated, which is the downsampling value of the block. The formula is as follows:

[0069] C i (h i ,w j )=(C i-1 (h i *2,w j *2)+C i-1 (h i *2,w j *2+1)+

[0070] C i-1 (h i *2+1,w j *2)+C i-1 (h i *2+1,w j *2+1)) / 4

[0071] P i (h i ,w j )=(P i-1 (h i *2,w j *2)+P i-1 (h i *2,w j *2+1)+

[0072] P i-1 (h i *2+1,w j *2)+P i-1 (h i *2+1,w j *2+1)) / 4

[0073]

[0074] Where i = 1,…,n-1, (h i ,w j ) is the hth i Row w j Column position, C i is the current image of the i-th layer of the pyramid, P i is the reference image of the jth level of the pyramid, is the motion vector prediction result output from step S3, It is the update result of the motion vector of the i-1th layer after filtering by the down-sampling Kalman filter model. The motion vectors are all two-dimensional vectors.

[0075] Then, step S5 is performed, according to the C i and P i , using dense optical flow to calculate the motion vector of each point in the current layer of the pyramid

[0076] In step S2 and step S5, the dense optical flow method is used to calculate the motion vector of the current i-th layer and i=1,…,n-1. The optical flow method assumes that the motion vectors in the local area are the same, and the spatial position of the point to be processed is (h x ,w y ), the local area centered on the point to be processed is [flow_size, flow_size], flow_size is set according to the hardware implementation cost and effect, and in this implementation example, flow_size=5. First, the horizontal gradient, vertical gradient and image pixel difference of each pixel in the local area are calculated. The value range is [h x -2,h x +2] points The value range is [w y -2,w y +2] points

[0077] The pixel difference is calculated by the following formula

[0078]

[0079] The calculation of the gradient formula uses the following operators:

[0080]

[0081]

[0082] Then the formulas for the horizontal gradient and vertical gradient of the current image are:

[0083]

[0084]

[0085] According to the optical flow formula, we have:

[0086]

[0087] There are 25 points in the 5x5 local area, so the following optical flow equations can be listed:

[0088]

[0089] where p i , p j The value range is -2 to 2, mv h ,mv w is the value of each dimension of the motion vector being sought.

[0090] The least squares method can be used to solve it, and the formula is as follows:

[0091]

[0092]

[0093]

[0094] Among them, if dm i =0, then mv_w = 0, mv_h = 0. Otherwise, use the following formula to calculate:

[0095]

[0096]

[0097] Then, step S6 is executed to establish a down-sampling Kalman filter model, using the motion vector The transformed observations are used to predict the motion vector Update to obtain the updated result of the motion vector of the i-th layer after filtering Specifically, first, according to the result of step S3 Establish a prediction model, then design a measurement matrix based on the optical flow model, and then calculate the Kalman gain. Transformed Observation Pair Get the updated result of the motion vector of the i-th layer after filtering

[0098] In this embodiment, a down-sampling Kalman filter model is established in step S6, assuming that the current layer is the i-th layer of the pyramid, i=1, ..., n-1. The motion vectors are all unified as 2x1 vectors, and the down-sampling Kalman filter model is as follows:

[0099] a) The state prediction equation is:

[0100]

[0101] The above process is the mean downsampling process in step S3.

[0102] b) Variance prediction equation. It is not difficult to derive the following equation from the state prediction equation:

[0103]

[0104] Among them, Q(h x ,w y ) is the 2x2 process noise covariance. Assuming that the downsampling process is an accurate process, the variance of the disturbance term is very small. Usually, Q(h x ,w y ) is a 0 vector. Depend on It is derived that in step e below, And by Derived from, is the variance matrix of the motion vector after Kalman filtering. Therefore, according to the recursive formula, it is only necessary to initialize the variance when i=0 to deduce it in sequence. The dimension is 2x2. Assuming that the two dimensions of the movement direction are independent of each other, then is a diagonal matrix, that is:

[0105]

[0106] in, are the variances of the two dimensions of the original amplitude motion vector, assuming that the current point is (h x ,w y ), the vertical and horizontal gradients of the current point are and According to the optical flow equation, the variance optical flow equation can be deduced as:

[0107]

[0108] Among them, varDif 0 (h x ,w y ) is the difference Dif between the current image pixel and the reference image pixel at the same position 0 (h x ,wy ) is the sum of the variance of the current image pixel and the variance of the reference image pixel at the same position.

[0109] Right now

[0110] varDif 0 (h x ,w y )=varC i (h x ,w y )+varP i (h x ,w y )

[0111] Using the assumption that the motion vectors in the local area are the same, the optical flow variance equation for each point in the local area is listed in the same way, and then the least squares method can be used to solve it. and Substitute the variance prediction equation to find the variance matrix of each point In the following step e, Find Therefore, after multiple layers of iteration, the i-th layer can be obtained.

[0112] c) Kalman gain equation is:

[0113]

[0114] Among them, GC i (h x ,w y ) is a 1x2 vector consisting of the horizontal and vertical gradients of the current point, that is:

[0115]

[0116] In the downsampling Kalman filter model of this embodiment, GC i (h x ,w y ) as the observation matrix H.

[0117] R(h x ,w y ) is the measurement noise covariance, with a dimension of 1x1, which is the error between the measured data and the true data, that is, vardif(h mentioned in step b x ,w y ), the final step c outputs the Kalman gain K of 2x1.

[0118] d) The state update equation is:

[0119]

[0120] e) The variance update equation is as follows:

[0121]

[0122] Then, step S7 is executed. Set the total number of pyramid layers to n, let i = i + 1. If i < n, repeat steps S3 - S6. If i = n, execute step S8 to obtain the updated result of the motion vector at the top layer of the filtered pyramid. In this embodiment, according to the downsampled Kalman filter model, Kalman filtering is performed on the motion vectors of the downsampled frame, and the obtained result is used as the input for the next layer of the frame. The Kalman filter model is continuously iterated until the motion vector result at the top layer is output.

[0123] Then, step S8 is executed. For the current image C at the top layer n-1 For each pixel point, using the updated result of the motion vector at the top layer as the center, a full - search block - matching algorithm is performed in the local area for the reference image P n-1 to obtain the motion vector of the block - matching search as the secondary updated result of the motion vector at the top layer. In this embodiment, according to the updated result of the motion vector at the top layer of the pyramid output in step S7 For each pixel point of the current image C at the top layer n-1 using the pixel position of the top - layer reference image P pointed to by its motion vector update result n-1 as the center, a full - search block - matching algorithm is performed within the local window of [s_size, s_size]. The size of the matching window is [m_size, m_size], and the motion vector corresponding to the best result of the block - matching is output as the secondary updated result. Among them, the values of s_size and m_size depend on the cost of hardware implementation. In this embodiment, s_size takes 5 and m_size takes 5, and the similarity calculation method for block - matching uses the Manhattan distance calculation.

[0124] Then, step S9 is executed. Initialize j = n - 2, and upsample each dimension of the secondary updated result of the motion vector at the (j + 1) - th layer to obtain the preliminary prediction result of the motion vector at the j - th layer.

[0125] Preferably, bilinear interpolation sampling is performed on each dimension of the secondary updated result of the motion vector at the (j + 1) - th layer to obtain the preliminary prediction result of the motion vector at the j - th layer, where the height and width are each twice the height and width of the original frame.

[0126] Then, step S10 is performed, for the current image C of the jth layer j For each pixel point, the motion vector of the jth layer is used to preliminarily predict the result The reference image P pointed to j The position of is taken as the center, and a block matching algorithm is performed in the local area to obtain the motion vector of the block matching search as the final prediction result of the motion vector of the jth layer. In this embodiment, a block matching algorithm is used to perform a full search within a local window of [s_size, s_size], and the size of the matching window is [m_size, m_size].

[0127] Then, step S11 is executed to establish an up-sampling Kalman filter model, using the update result of the motion vector of the jth layer The final prediction result of the transformed observation on the motion vector of the jth layer Update to obtain the secondary update result of the motion vector of the jth layer after filtering Specifically, according to the results of step S6 and step S10, an upsampling Kalman filter model of motion vector is established. First, according to the result of step S10, Establish a prediction model, then design a measurement matrix based on the optical flow model, and then calculate the Kalman gain. The final prediction result of the transformed observation on the motion vector of the jth layer Update. Obtain the secondary update result of the motion vector of the jth layer after filtering

[0128] In this embodiment, the up-sampling Kalman filter model in step S11 is constructed in a manner similar to the down-sampling Kalman filter model in step S6, specifically as follows:

[0129] a) The state prediction equation is changed from the prediction equation of mean downsampling to the prediction equation of bilinear interpolation upsampling, that is:

[0130]

[0131]

[0132] Wherein bilinear() is a bilinear interpolation algorithm, and the horizontal and vertical upsampling are 2 times, that is, the process of step S9.

[0133] block_mc() is The pointed position is taken as the center, and the optimal motion vector is found locally using a block matching algorithm, which is the process of step S10.

[0134] b) In the variance prediction equation, the top-level variance The variance of the top layer of the downsampled Kalman filter model is approximate.

[0135] In the process of bilinear interpolation, it can be deduced that the motion vector of each point in the upsampled image is the weighted coefficient of the four motion vectors corresponding to the current image, denoted as w1 j ,w2 j ,w3 j ,w4 j , the variances of these four motion vectors are recorded as

[0136] Then the variance of the motion vector upsampled by bilinear interpolation is , where the formula for each dimension is as follows:

[0137]

[0138]

[0139] To simplify the calculation, the variance is assumed to remain unchanged during the block matching process. Therefore, the variance prediction equation of the upsampling Kalman filter is the above formula.

[0140] c) The Kalman gain equation is the same as that in the downsampled Kalman filter model, namely:

[0141]

[0142] d) The state update equation changes the motion vector of the input and output, namely:

[0143]

[0144] e) The formula of the variance update equation is the same as that in the downsampling Kalman filter model, namely:

[0145]

[0146] Finally, execute step S12, set j = j-1, if j ≥ 0, repeat steps S9-S11, if j < 0, then is the final result of the motion vector estimation, that is, the motion vector estimation result of the current image to be processed.

[0147] Based on this, an embodiment of the present invention further provides a motion vector estimation system for implementing the motion vector estimation method as described above. Figure 2 As shown, including:

[0148] The data input unit 100 is configured to input the current image C to be processed. 0 And the reference image P0 ;

[0149] The first calculation unit 101 is configured to calculate the update result of the motion vector of each point in the current layer of the pyramid using the dense optical flow method And initialize i=1;

[0150] The motion vector downsampling unit 102 is configured to update the motion vector of the i-1th layer Downsample each dimension to obtain the motion vector prediction result of the i-th layer

[0151] The image downsampling unit 103 is configured to downsample the current image C of the i-1th layer i-1 and the reference image P i-1 Downsample to obtain the current image C of the i-th layer i and the reference image P i ;

[0152] The second calculation unit 104 is configured to calculate the i and P i , using dense optical flow to calculate the motion vector of each point in the current layer of the pyramid

[0153] The down-sampling Kalman filter unit 105 is configured to establish a down-sampling Kalman filter model using the motion vector The transformed observations are used to predict the motion vector Update to obtain the updated result of the motion vector of the i-th layer after filtering

[0154] The third computing unit 106 is configured to calculate the topmost current image C n-1 For each pixel point, the top-level motion vector is used to update the result The reference image P pointed to n-1 The position of is taken as the center, and a block matching algorithm is performed in a local area to perform a full search, and the motion vector of the block matching search is obtained as the secondary update result of the motion vector of the top layer

[0155] The motion vector upsampling unit 107 is configured to initialize j=n-2, and to update the motion vector of the j+1th layer twice. Upsample each dimension to obtain the preliminary prediction result of the motion vector of the jth layer

[0156] The fourth computing unit 108 is configured to calculate the current image C of the jth layer. j For each pixel point, the motion vector of the jth layer is used to preliminarily predict the result The reference image P pointed to j The position of is taken as the center, and a block matching algorithm is performed in the local area to obtain the motion vector of the block matching search as the final prediction result of the motion vector of the jth layer.

[0157] The up-sampling Kalman filter unit 109 is configured to establish a down-sampling Kalman filter model, using the update result of the motion vector of the jth layer The final prediction result of the transformed observation on the motion vector of the jth layer Update to obtain the secondary update result of the motion vector of the jth layer after filtering

[0158] In this embodiment, the first calculation unit 101 is used to calculate the horizontal gradient, vertical gradient and pixel difference of each position in the local area of ​​the point to be processed and the point at the same position in the reference image, and establish a dense optical flow equation group, and use the least squares method to solve the initial value of the motion vector of the original frame.

[0159] In this embodiment, the motion vector downsampling unit 102 uses a mean downsampling method to divide the input motion vector into different 2x2 blocks, and takes the mean of each dimension of the four motion vectors of each 2x2 block to obtain the motion vector of the corresponding downsampling amplitude. If the current layer is the bottom layer of the pyramid, the input motion vector is the output of the first calculation unit 101, otherwise, the input motion vector is the output of the downsampling Kalman filter unit 105.

[0160] In this embodiment, the image downsampling unit 103 adopts a mean downsampling method to divide the current image and the reference image of the input current layer into different 2x2 blocks, and average the four pixel values ​​of each 2x2 block to obtain the pixel values ​​of the downsampling format corresponding to the current image and the pixel values ​​of the downsampling format corresponding to the reference image. If the current layer is the bottom layer of the pyramid, the input current image and the reference image are the current image C 0 And the reference image P 0 , otherwise it is the result of its most recent output.

[0161] In this embodiment, the second calculation unit 104 is used to calculate the horizontal gradient, vertical gradient and pixel difference of each position in the local area of ​​the point to be processed on the current sampling frame, and establish a dense optical flow equation group, and use the least squares method to solve the motion vector of the current sampling frame.

[0162] In this embodiment, the down-sampling Kalman filter unit 105 uses the motion vector output by the second calculation unit 104 and the motion vector output by the motion vector down-sampling unit 102 to establish a down-sampling Kalman filter model, and outputs the result of the motion vector of the current layer after the Kalman filter. If the current layer is the top layer of the pyramid, the result is output to the third calculation unit 106, otherwise the result is output to the motion vector down-sampling unit 102.

[0163] In this embodiment, the third calculation unit 106 is used to update the block matching result of the motion vector at the top layer of the pyramid output by the downsampling Kalman filter unit 105 according to the position of the top layer of the reference image pointed to by the motion vector at the top layer of the pyramid, and use a full search block matching algorithm in a local area with the position as the center.

[0164] In this embodiment, the motion vector upsampling unit 107 is used to perform an upsampling operation on the input motion vector, and the upsampling method is to perform bilinear interpolation on each dimension of the motion vector to obtain a motion vector result that is upsampled by 2 times in both horizontal and vertical directions. If the current layer is the top layer of the pyramid, the input motion vector is the output of the third calculation unit 106, otherwise, the input motion vector is the output of the upsampling Kalman filter unit 109.

[0165] In this embodiment, the fourth calculation unit 108 is used to obtain the motion vector of the block matching method of the current layer by using a full-search block matching algorithm in a local area centered on the position of the reference image of the current layer pointed to by the motion vector output by the motion vector upsampling unit 107.

[0166] In this embodiment, the up-sampling Kalman filter unit 109 is used to establish an up-sampling Kalman filter model based on the motion vector output by the fourth calculation unit 108 and the motion vector output by the down-sampling Kalman filter unit 105, and outputs the result of the motion vector of the current layer after the Kalman filter. If the current layer is the bottom layer of the pyramid, the final motion vector result of the current layer is output, otherwise it is output to the motion vector up-sampling unit 107.

[0167] Based on the same inventive concept, an embodiment of the present invention further provides a readable storage medium having a computer program stored thereon, and when the computer program is executed, the motion vector estimation method as described above can be implemented.

[0168] The readable storage medium can be a tangible device that can keep and store the instructions used by the instruction execution device, such as but not limited to an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device or any suitable combination thereof. The more specific example (non-exhaustive list) of the readable storage medium includes: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a convex structure in a groove on which instructions are stored, and any suitable combination thereof. The computer program described herein can be downloaded from the readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network and / or a wireless network. The network can include copper transmission cables, optical fiber transmission, wireless transmission, routers, firewalls, switches, gateway computers and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer program from the network and forwards the computer program for storage in the readable storage medium in each computing / processing device. The computer program for performing the operation of the present invention can be a run instruction, an instruction set architecture (ISA) instruction, a machine instruction, a machine-related instruction, a microcode, a firmware instruction, a state setting data, or a source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as "C" language or similar programming languages. The computer program can be executed completely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, using an Internet service provider to connect through the Internet). In some embodiments, by utilizing the state information of a computer program to personalize an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), the electronic circuit can execute computer-readable program instructions to implement various aspects of the present invention.

[0169] Here, various aspects of the present invention are described with reference to the flowchart and / or block diagram of the method, system and computer program product according to the embodiment of the present invention. It should be understood that each square frame of the flowchart and / or block diagram and the combination of the square frames in the flowchart and / or block diagram can be realized by a computer program. These computer programs can be provided to the processor of a general-purpose computer, a special-purpose computer or other programmable data processing device, so as to produce a machine, so that these programs, when executed by the processor of a computer or other programmable data processing device, produce a device for realizing the function / action specified in one or more square frames in the flowchart and / or block diagram. These computer programs can also be stored in a readable storage medium, and these computer programs make the computer, programmable data processing device and / or other equipment work in a specific way, so that the readable storage medium storing the computer program includes a manufactured product, which includes instructions for realizing various aspects of the function / action specified in one or more square frames in the flowchart and / or block diagram.

[0170] The computer program may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are executed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the computer program executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0171] In summary, the embodiments of the present invention provide a motion vector estimation method, system and readable storage medium, which improve the block matching algorithm of the pyramid structure, establish prediction equations and update equations of motion vectors between adjacent layers, and perform Kalman filtering on the motion vectors layer by layer to correct the motion vectors, thereby solving the problem that the erroneous motion vectors calculated by the block matching of the traditional pyramid structure will be transmitted layer by layer; at the same time, an optical flow algorithm of a downsampling pyramid structure is additionally established, which still uses Kalman filtering to update the motion vectors layer by layer from the bottom layer to the top layer, and uses full-search block matching to update the top layer locally with the updated motion vector as the center, thereby solving the problem that the calculation accuracy of the top-level motion vector of the block matching of the traditional pyramid structure is poor, so that the motion vector finally output is clean and relatively accurate.

[0172] The above description is only a description of the preferred embodiment of the present invention, and is not intended to limit the scope of the present invention. Any changes and modifications made by a person skilled in the art in the field of the present invention based on the above disclosure are within the scope of protection of the present invention. Obviously, a person skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations fall within the scope of the present invention and its equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A motion vector estimation method, characterized in that: include: S1, obtaining the current image C0 and the reference image P0 to be processed; S2. Use the dense optical flow method to calculate the updated result of the motion vector of each point in the current layer of the pyramid Initialize i=1; S3. Update result of motion vector of i-1th layer Downsample each dimension to obtain the motion vector prediction result of the i-th layer S4, for the current image C of the i-1th layer i-1 and the reference image P i-1 Downsample to obtain the current image C of the i-th layer i and a reference image Pi; S5. According to the C i and P i , using dense optical flow to calculate the motion vector of each point in the current layer of the pyramid S6, establish a down-sampling Kalman filter model, using the motion vector The transformed observations are the motion vector prediction results Update to obtain the updated result of the motion vector of the i-th layer after filtering S7. Set the total number of pyramid layers to n, let i = i + 1. If i < n, repeat steps S3 - S6. If i = n, execute step S8 to obtain the updated result of the motion vector at the top layer of the filtered pyramid. S8. For the topmost current image C n-1 For each pixel point, the top-level motion vector is used to update the result The reference image P pointed to n-1 The position of is taken as the center, and a block matching algorithm is performed in a local area to perform a full search, and the motion vector of the block matching search is obtained as the secondary update result of the motion vector of the top layer S9, initialize j=n-2, and update the motion vector of the j+1th layer twice Upsample each dimension to obtain the preliminary prediction result of the motion vector of the jth layer S10, for the current image C of the jth layer j For each pixel point, the motion vector of the jth layer is used to preliminarily predict the result The reference image P pointed to j The position of is taken as the center, and a block matching algorithm is performed in the local area to obtain the motion vector of the block matching search as the final prediction result of the motion vector of the jth layer. S11, establish an upsampling Kalman filter model, using the update result of the motion vector of the jth layer The final prediction result of the transformed observation on the motion vector of the jth layer Update to obtain the secondary update result of the motion vector of the jth layer after filtering S12, let j = j-1, if j ≥ 0, then repeat steps S9-S11, if j < 0, then is the final result of motion vector estimation.

2. The motion vector estimation method according to claim 1, characterized in that: The reference image is an image of a previous frame of the current image after being subjected to spatiotemporal denoising processing.

3. The motion vector estimation method according to claim 1, characterized in that: In step S3 and step S4, the downsampling method adopted is 2x2 mean downsampling.

4. The motion vector estimation method according to claim 1, characterized in that: In step S9, the upsampling method adopted is bilinear interpolation upsampling.

5. The motion vector estimation method according to claim 1, characterized in that: In step S6, when establishing the down-sampling Kalman filter model of the motion vector, firstly according to the result of step S3 Design a prediction model, then design a measurement matrix based on the optical flow algorithm, and then calculate the Kalman gain. The transformed observations are to update.

6. The motion vector estimation method according to claim 1, characterized in that: In step S11, when establishing the up-sampling Kalman filter model of the motion vector, firstly according to the result of step S10 Establish a prediction model, then design a measurement matrix based on the optical flow algorithm, and then calculate the Kalman gain. The transformed observations update the motion vectors.

7. A motion vector estimation system, used to implement the motion vector estimation method according to any one of claims 1 to 6, characterized in that: include: A data input unit, configured to input a current image C0 to be processed and a reference image P0; The first calculation unit is configured to calculate the update result of the motion vector of each point in the current layer of the pyramid using the dense optical flow method And initialize i=1; The motion vector downsampling unit is configured to update the motion vector of the i-1th layer Downsample each dimension to obtain the motion vector prediction result of the i-th layer The image downsampling unit is configured to downsample the current image C of the i-1th layer i-1 and the reference image P i-1 Downsample to obtain the current image C of the i-th layer i and a reference image Pi; The second computing unit is configured to calculate the i and Pi, using dense optical flow to calculate the motion vector of each point in the current layer of the pyramid The down-sampling Kalman filter unit is configured to establish a down-sampling Kalman filter model using the motion vector The transformed observations are the motion vector prediction results Update to obtain the updated result of the motion vector of the i-th layer after filtering The third computing unit is configured to calculate the current image C at the top layer. n-1 For each pixel point, the top-level motion vector is used to update the result The reference image P pointed to n-1 The position of is taken as the center, and a block matching algorithm is performed in a local area to perform a full search, and the motion vector of the block matching search is obtained as the secondary update result of the motion vector of the top layer The motion vector upsampling unit is configured to initialize j=n-2 and update the motion vector of the j+1th layer twice Upsample each dimension to obtain the preliminary prediction result of the motion vector of the jth layer The fourth computing unit is configured to calculate the current image C of the jth layer. j For each pixel point, the motion vector of the jth layer is used to preliminarily predict the result The reference image P pointed to j The position of is taken as the center, and a block matching algorithm is performed in the local area to obtain the motion vector of the block matching search as the final prediction result of the motion vector of the jth layer. The upsampling Kalman filter unit is configured to establish an upsampling Kalman filter model using the updated result of the motion vector of the jth layer The final prediction result of the transformed observation on the motion vector of the jth layer Update to obtain the secondary update result of the motion vector of the jth layer after filtering 8. The motion vector estimation system according to claim 7, characterized in that: For the motion vector downsampling unit, if the current layer is the bottom layer of the pyramid, the input motion vector is the output of the first calculation unit, otherwise, the input motion vector is the output of the downsampling Kalman filter unit.

9. The motion vector estimation system according to claim 7, characterized in that: For the motion vector upsampling unit, if the current layer is the topmost layer of the pyramid, the input motion vector is the output of the third calculation unit, otherwise, the input motion vector is the output of the upsampling Kalman filter unit.

10. A readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed, the motion vector estimation method according to any one of claims 1 to 6 can be implemented.

Images