Optical flow estimation method using video coding information in combination with attention and occlusion prediction

Abstract

The invention relates to the technical field of optical flow estimation in the field of computer vision, in particular to an optical flow estimation method combining video coding information with attention and occlusion prediction, which comprises the following steps of: at a video receiving end, receiving compressed video coding information, extracting motion vector, residual error and coding block information in the coding information, and calculating the video coding information according to the motion vector, residual error and coding block information; converting the image information into image information; by utilizing an attention mechanism, extracting cross correlation between the motion vector and residual error information, filling discrete motion vectors with residual errors, and correcting wrong motion vectors; coding block information is used as main prior information, the motion vector and residual information obtained in the second step after attention calculation are added, and a predicted occlusion area is obtained through an occlusion prediction module; and through a motion field conversion model, fusing the obtained predicted occlusion area with the motion vector and the residual error again to obtain a final estimated optical flow. According to the algorithm provided by the invention, the acquisition difficulty of the optical flow in practical application is greatly reduced.

Description

Technical Field

[0001] This invention relates to the field of optical flow estimation technology in computer vision, specifically to an optical flow estimation method that utilizes video coding information combined with attention and occlusion prediction. Background Technology

[0002] Optical flow has important applications in motion recognition, object detection, and other fields. Optical flow calculation typically uses information from the original video frames to calculate the displacement between corresponding points. However, during transmission, video is usually stored as encoded video information. Therefore, in practical applications, calculating optical flow usually requires first decoding the compressed video information back into the original video frames. This decoding process is very time-consuming, thus affecting the real-time performance of many tasks. Furthermore, motion occlusion is a key factor affecting estimation accuracy. When objects occlude each other, the assumptions of brightness consistency and spatial smoothness underlying traditional optical flow estimation methods are violated. Motion vectors in video compression are similar to optical flow, both representing motion information between two frames. Directly using motion vectors from encoded video information to estimate optical flow can reduce the time complexity of optical flow calculation, but it leads to a decrease in accuracy. In this context, existing optical flow estimation methods using original video frames cannot adequately meet the real-time requirements of practical applications. Therefore, if we can utilize motion vector information from encoded video information and combine it with other information in the encoded information to enhance the motion representation of the motion vectors, we can improve the speed of optical flow estimation while maintaining accuracy.

[0003] After Horn and Schunk (HS) proposed the first variational optical flow estimation framework, optical flow became an important research direction in computer vision. Following Horn, Lucas and Kanade (LK) introduced local constraints and estimated sparse flow fields. Many scholars have made numerous improvements and modifications to the variational optical flow method based on the basic methods of HS and LK. With the development of deep learning, due to its computational efficiency, some scholars have begun to use deep learning methods for optical flow estimation. After Dosovitskiy proposed the first deep learning-based optical flow estimation network, FlowNet, many scholars have begun to try to obtain optical flow more quickly and efficiently based on deep learning methods. Although current deep learning methods have improved the speed of optical flow estimation, most deep learning optical flow methods, like traditional variational methods, require the original video frames as input, resulting in high time complexity at the decoding end. The MV2Flow method uses motion vectors and residuals in the encoded information, combined with some original video frame information, for optical flow estimation. However, since it still needs to add some original video frames, the complexity needs to be further reduced. S. Zhou used motion vectors and a portion of the original frames, combined with an attention method, for optical flow estimation. However, the accuracy of the results needs further improvement because the influence of occlusion areas was not considered. Currently, there is no fast optical flow estimation algorithm based on deep learning that fully utilizes video coding information. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention proposes an optical flow estimation method that combines video coding information with attention and occlusion prediction, based on the relationship between video coding information (motion vectors, residuals, and coding block information) and optical flow.

[0005] To achieve the above objectives, the following technical solution is provided:

[0006] A method for estimating optical flow using video coding information combined with attention and occlusion prediction, characterized by comprising the following steps:

[0007] Step 1: At the video receiving end, receive the compressed video encoding information, extract the motion vectors, residuals, and coded block information from the encoding information, and convert them into image information;

[0008] Step 2: Utilize the attention mechanism to extract the cross-correlation between motion vectors and residual information, so that the residuals fill the discrete motion vectors and correct erroneous motion vectors;

[0009] Step 3: Using the coding block information as the main prior information, and adding the motion vector and residual information obtained in Step 2 after attention calculation, the predicted occlusion area is obtained through the occlusion prediction module.

[0010] Step 4: Using the adversarial generative motion field transformation model based on U-net, the predicted occlusion region is fused with the motion vector and residual again to obtain the final estimated optical flow.

[0011] Preferably, in step two, the attention mechanism (attention calculation method) includes a self-attention calculation and a mutual attention calculation. First, a self-attention calculation is used to enhance the spatial correlation between the motion vector and the residual. However, the mutual correlation between the motion vector and the residual cannot be extracted by self-attention calculation alone. Therefore, after the self-attention calculation, a mutual attention calculation is performed on the motion vector and the residual to fill the discrete motion vector with the residual and correct the erroneous motion information in the motion vector.

[0012] The method for calculating self-attention is as follows:

[0013] Q MV1 =E A (MV),K MV1 =E A (MV),V MV1 =E A (MV) (1)

[0014] Q RES1 =E B (RES),K RES1 =E B (RES),V RES1 =E B (RES) (2)

[0015]

[0016] Where MV and RES represent the motion vector and residual, respectively, and E A and E B Q represents the feature encoding process. MV1 K MV1 V MV1 Q RES1 K RES1 and V RES1 This represents the feature vector after feature encoding, where B represents the short-distance positional offset; using To ensure that the variance of the dot product remains 1 regardless of the vector length, Att MV1 and Att RES1 This represents the final attention calculation result;

[0017] The method for calculating mutual attention is as follows:

[0018] Q MV2 =E C (ATT MV1 ),K MV2 =EC (ATT MV1 ),V MV2 =E C (ATT RES1 (5)

[0019]

[0020] Among them, E C Att represents the feature encoding process. MV2 This represents the motion vector information after residual correction.

[0021] Preferably, in step three, the occlusion prediction module uses image reconstruction loss as the loss function, and the calculation method is as follows:

[0022]

[0023] Among them, L OMP denoted by , n represents the total loss of occlusion prediction, n represents the total number of pixels in the input information, f(I) represents the actual occlusion information used as supervision information, and I' represents the predicted occlusion information.

[0024] Preferably, in step four, after obtaining the corrected motion vectors, residuals, and occlusion predictions, they are input into a U-net-based adversarial generative motion field transformation model. This model includes a U-net-based optical flow generator and an optical flow discriminator to obtain the final dense optical flow. The average endpoint error loss (EPE) is used as the optical flow field reconstruction loss function; its definition is as follows:

[0025]

[0026] Among them G OF-LIKE (X) represents the optical flow-like flow generated by the network, G OF (X) represents the actual optical flow as guiding information, and LOSS is the optical flow reconstruction loss.

[0027] While using motion vectors and residuals alone for optical flow estimation can achieve good results in most areas, it cannot handle optical flow in occluded regions. To address this issue, this invention adds an occlusion prediction module to provide occlusion region information for optical flow estimation. Since intra-coded blocks are strongly correlated with occlusion regions, they are used as prior information for occlusion prediction. Because intra-coded blocks do not contain additional contour and motion information, we add the motion vectors and residuals obtained in the previous step as additional inputs. This information has already been calculated and incurs no additional computational cost.

[0028] The basic ideas of this invention can be summarized in the following three points:

[0029] 1. Directly utilize video coding information to improve the speed of optical flow estimation. During video transmission, motion vector information generated in the coding information can be obtained with very low computational cost. Furthermore, motion vectors, like optical flow, can represent motion information between two frames. Using motion vectors for optical flow estimation can save the time of decoding the original video frames, greatly reducing the computational complexity of obtaining optical flow.

[0030] 2. Compared to optical flow, motion vectors are discrete fields while optical flow is a dense field. Using only discrete motion vectors for optical flow estimation cannot achieve high accuracy. In bitstream information, the residual represents the difference between the predicted frame obtained from the motion vector and the original frame; therefore, the residual is usually large when the motion vector is large. Adding the residual as an additional input can fill in the discrete motion vector and correct errors in the motion vector while maintaining low complexity. Attention methods can effectively extract the cross-correlation between various information components.

[0031] 3. Motion occlusion regions typically appear in the previous frame but are obscured due to object movement. This affects the accurate calculation of optical flow in optical flow estimation. In video coding, frames using inter-frame coding will encode the portion of the target lost due to motion occlusion using intra-frame coded blocks. Therefore, intra-frame coded blocks in inter-frame coded frames have a strong correlation with occlusion regions. Thus, they can serve as reliable priors for occlusion region prediction.

[0032] The beneficial effects of this invention are as follows:

[0033] 1. This invention fully utilizes video coding information, eliminating the time required to decode bitstream information into raw frames, which is common in most deep learning-based optical flow methods. Based on the relationship between motion vectors and residuals in the bitstream information, this invention uses attention calculation to extract the cross-correlation between motion vectors and residuals, allowing the residuals to fill in discrete motion vectors and correct erroneous motion information. Furthermore, based on the correlation between occlusion regions and intra-frame coded blocks, this invention adds intra-frame coded block information as prior information for occlusion region prediction, thus predicting the occlusion region.

[0034] 2. This invention utilizes block-based motion vector information from the compressed domain for optical flow estimation. This avoids decoding the compressed domain information into the original video frame, saving time and reducing system complexity. To address the significant difference between the motion vector field and the optical flow field, this invention adds residual information, also used as video coding information, as an additional input. Utilizing the relationship between the residual information and the motion vector, attention calculations are used to fill and correct the motion vector, narrowing the gap between the motion vector and the optical flow. For motion occlusion, this invention designs an occlusion region prediction module based on the relationship between intra-frame coded blocks and occlusion regions, providing the correct occlusion region for optical flow estimation, thereby improving the accuracy of optical flow estimation in occluded regions. Compared to methods using original video frames to calculate optical flow, the algorithm proposed in this invention significantly reduces the difficulty of obtaining optical flow in practical applications. Because residual information is added to fill and correct the motion vector, the resulting optical flow information is denser, with fewer erroneous motion information. Simultaneously, using coded block information as prior information for occlusion region prediction also achieves good prediction results in occluded regions. Attached Figure Description

[0035] Figure 1 This is a comparison between the optical flow estimation map obtained by the algorithm of this invention and the actual optical flow. Detailed Implementation

[0036] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0037] Example 1

[0038] The algorithm of this invention obtains a conversion model through a training process, and the process of converting video encoded information into optical flow is as follows:

[0039] 1. Convert the training videos into video encoding information, and then directly convert the obtained encoding information into images as the training dataset;

[0040] 2. Using the obtained motion vectors, residuals, and coding block information as the training dataset, the model of this invention is trained using a deep learning environment built under PyCharm to obtain the required conversion model;

[0041] 3. Convert the bitstream information used for testing into motion vectors, and convert the residual and coded block information into images;

[0042] 4. Convert motion vectors, residuals, and coded block information into images and directly input them into the trained model to obtain the estimated optical flow;

[0043] 5. Statistical results show that, among the three widely used optical flow datasets (SPI, KITTI 2012, and KITTI 2015), the method of this invention has lower time complexity compared to the method using the original frames. This fully demonstrates the efficiency of the method of this invention.

[0044] 6. Table 1 compares the time complexity of obtaining the final optical flow from video coding information using the present invention and other advanced methods in different datasets, with 10 frames as a group.

[0045] Table 1 compares the time complexity of this invention with various state-of-the-art methods on different datasets.

[0046]

[0047] Among them, RAFT, GMA, CRAFT, and GMFLOW are optical flow estimation methods that use video frames, MVFlow is an optical flow estimation method that uses motion vectors from partial video frames plus video coding information, and this method is an optical flow estimation method that uses video coding information entirely. SPI, KITTI 2012, and KITTI 2015 are the three most widely used optical flow estimation datasets, so we chose to verify the time complexity of this invention on these three datasets.

[0048] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

Claims

1. A method for optical flow estimation using video coding information combined with attention and occlusion prediction, characterized in that, The method comprises the following steps: Step one, receiving compressed video encoding information at the video receiving end, extracting motion vector, residual error and encoding block information in the encoding information, and converting them into image information; Step two, using attention mechanism to extract the cross-correlation of motion vector and residual error information, so as to fill the discrete motion vector with residual error and correct the error motion vector; Step three, using the encoding block information as the main prior information, adding the motion vector and residual error information obtained in step two after attention calculation, and obtaining the predicted occlusion area through the occlusion prediction module; Step four, fusing the obtained predicted occlusion area with the motion vector and residual error again through the motion field conversion model to obtain the final estimated optical flow.

2. The optical flow estimation method using video coding information combined with attention and occlusion prediction according to claim 1, characterized in that, In step two, the attention mechanism comprises a self-attention calculation and a cross-attention calculation. First, the self-attention calculation is used to enhance the spatial correlation of the motion vector and the residual error, and then the cross-attention calculation is used to fill the discrete motion vector with the residual error and correct the error motion information in the motion vector.

3. The optical flow estimation method using video coding information combined with attention and occlusion prediction according to claim 2, characterized in that, The method of the self-attention calculation is as follows: Q MV1 = E A (MV), K MV1 = E A (MV), V MV1 = E A (MV) (1) Q RES1 = E B (RES), K RES1 = E B (RES), V RES1 = E B (RES) (2) where MV and RES represent the motion vector and the residual, E A and E B denote the feature encoding process, Q MV1 , K MV1 , V MV1 , Q RES1 , K RES1 and V RES1 denote the feature vector after feature encoding, B denotes the short distance position offset; using to ensure that the variance of the dot product is still 1 regardless of the length of the vector, Att MV1 and Att RES1 denote the final attention calculation result.

4. The optical flow estimation method using video coding information combined with attention and occlusion prediction according to claim 3, characterized in that, The method of the cross-attention calculation is as follows: Q MV2 = E C (ATT MV1 ), K MV2 = E C (ATT MV1 ), V MV2 = E C (ATT RES1 ) (5) wherein E C represents a feature encoding process, Att MV2 represents the final motion vector information after residual correction.

5. The optical flow estimation method using video coding information combined with attention and occlusion prediction according to claim 4, characterized in that, In step three, the occlusion prediction module uses image reconstruction loss as the loss function, and the calculation method is as follows: where L OMP represents the total loss of occlusion prediction, n represents the total number of pixels of input information, f(I) represents the real occlusion information as supervision information, and I' represents the predicted occlusion information.

6. The optical flow estimation method using video coding information combined with attention and occlusion prediction according to claim 5, characterized in that, In step four, after obtaining the corrected motion vector, residual error and occlusion prediction, they are input into the U-net-based adversarial generated motion field conversion model. The model comprises a U-net-based optical flow generator and an optical flow discriminator to obtain the final dense optical flow. The average endpoint error loss (EPE) is used as the optical flow field reconstruction loss function, and the definition is as follows: where LOSS is the optical flow reconstruction loss G OF-LIKE (X) represents the network-generated optical flow-like flow, G OF (X) represents the true optical flow as guiding information.

Images