An optical flow estimation method based on multi-scale global cross matching

Abstract

The application provides a light flow estimation method based on multi-scale global cross matching, comprising the following steps: 1. constructing an image feature enhancement network based on a multi-scale cross attention module (MCA); 2. constructing a light flow estimation module; 3. constructing a pixel processing module for an occlusion area; 4. inputting two continuous image frames at the input end of the network for supervised training; 5. inputting two continuous image frames in the trained model for testing, and the output is corresponding estimated light flow. The application proposes a multi-scale cross attention module (MCA), which uses the MCA module to complete the relevant information between different image blocks in the same feature image, so that the network can learn image information of multiple resolutions. At the same time, the pixel occlusion problem is solved by modeling the self-similarity of the image, and then the final predicted light flow is obtained.

Description

Technical Field

[0001] This invention provides an optical flow estimation method based on multi-scale global cross-matching, belonging to the field of computer vision. Background Technology

[0002] Optical flow estimation of consecutive video frames is a long-standing, fundamental, and challenging problem in computer vision. It is crucial for building higher-level cognitive abilities in scene understanding, such as object recognition, object tracking, action recognition, and scene segmentation. With the development of deep learning, using deep learning to solve the optical flow estimation problem has gradually become the mainstream approach. Recently, with the rise of Transformer in vision tasks, self-attention and the Transformer architecture have achieved success in several visual subtasks and have been applied to optical flow estimation to some extent. However, due to the massive computational cost of Transformer, image partitioning is often used to reduce computational cost, but this destroys the advantage of Transformer in establishing long-distance dependencies and loses the information correlation between the partitioned image patches. Therefore, this invention proposes a multi-scale cross-attention module, which uses multiple parallel cross-attention layers to extract features between image pairs at different scales. Cross-attention is calculated for the entire image features at low resolution and fine-grained cross-attention is calculated at high resolution. Then, the correlations are fused, enabling the network to obtain more complete feature correlations.

[0003] Occlusion is a major challenge in optical flow estimation. This invention adopts the modeling of image self-similarity, assuming that by finding points with similar appearance in the reference frame, the network can find points with similar motion. Taking advantage of the structural similarity between the optical flow image and the input image, an occlusion processing module composed of self-attention is designed. Summary of the Invention

[0004] This invention provides an optical flow estimation method based on multi-scale global cross-matching. It proposes a multi-scale cross-attention module (MCA) and constructs a feature enhancement network composed of multi-scale cross-attention modules. The MCA module is used to complete the relevant information between different image blocks in the same feature image, so that the network can learn multi-scale image information. Then, by taking advantage of the structural similarity between the optical flow image and the input image, the self-similarity of the image is modeled to solve the pixel occlusion problem, thereby improving the accuracy of optical flow estimation.

[0005] The purpose of this invention is achieved as follows: (1) Constructing an image feature enhancement network based on a multi-scale cross attention module. This model includes a feature extraction convolutional network and a multi-scale feature matching network composed of MCA modules. First, the input image is subjected to convolution operation to extract image features. Then, the feature image is encoded in position and input into the multi-scale feature matching network composed of MCA modules. The entire multi-scale feature matching network contains N MCA modules. Each MCA module performs correlation calculation on three different resolutions of the input image. The three different resolution image pairs are obtained by downsampling. After passing through the attention layer, they are restored to a unified resolution by upsampling and fused. Then, they are added to the input and output.

[0006] (2) Construct an optical flow estimation module; input the two frames of image features output from the last layer of the multi-scale feature matching network into the optical flow estimation module for prediction. This module consists of a global matching module and a softmax layer; perform dot product operation on the input features to obtain global correlation, and then normalize the last two dimensions of global correlation through softmax to obtain the matching probability. Then, multiply the matching probability by the weighted average of the 2D coordinates of the pixel grid points to obtain the correspondence matrix; finally, the optical flow is obtained by calculating the coordinate difference between corresponding points.

[0007] (3) Construct a pixel processing module for occlusion areas; this module consists of a self-attention layer, which inputs the 2D optical flow output by the optical flow estimation module and the target map features output by the multi-scale feature matching network into the self-attention layer, and then adds them to the 2D optical flow to obtain the final optical flow;

[0008] (4) Input two consecutive frames of images into the network input terminal and perform supervised training;

[0009] (5) Input two consecutive frames of images into the trained model for testing, and output the corresponding estimated optical flow.

[0010] The present invention also includes the following structural features:

[0011] 1. In feature (1), two consecutive frames of image I t and I t+1 First, the input is obtained from a feature extraction network composed of convolutions. The feature map is defined as follows: H and W represent the height and width of the input image, respectively, and C represents the number of channels in the input image. The feature map is then downsampled to H. l ×W l ×C, where H l With W l These are the height and width downsampled to the lowest resolution. They are manually specified constant values ​​that change depending on the resolution of the input image. After positional encoding, they are input to a cross-attention layer, whose attention operation is defined as follows:

[0012]

[0013] Where CAtt represents cross-attention computation; Q, K, and V represent the linear projections of the input features, specifically Q from the source features. Export, F s K and V represent the source features, i represents the feature index at different times; K and V represent the target features. Export F t Represents the target features, where i represents the feature index at different times; T represents the matrix transpose operation, softmax(·) represents the normalization operation, and D represents the Q and K dimensions; W q W k W v Representing three distinct parameter matrices, the global cross-attention score M is used to update the source features through the FFN layer of a feedforward neural network. The update operation is defined as follows:

[0014]

[0015] Where Cat represents the Concat operation, the updated output characteristics. and The image is then upsampled to the input size and then fed into a multi-scale image feature matching network consisting of MCA modules.

[0016] 2. The MCA module in feature (1) obtains three different resolution image feature pairs through downsampling, which are H from low to high resolution respectively. l ×W l ×C、 as well as The three feature resolutions correspond to an attention score of M. c M m and M f For the Kth MCA block, at a resolution of (H l W l For the feature image pairs, pixel-wise cross-attention calculation is performed. For the other two resolution features, they are first divided, and the original feature F is then processed. S i The image feature blocks are divided into S×S patches, and a total of 10 patches are formed. Then input the cross-attention module, where M c M m and M f The calculation method is the same as the M calculation method described above; the features output by each attention layer are upsampled to restore the input size, and then merged and compressed along the channels to obtain the result. Then, after passing through a single-layer convolutional network, the output is processed as follows:

[0017]

[0018] Where LN represents the normalization layer, ConV represents the convolutional layer, and Cat represents the Concat operation; The output features of the cross-attention module represent feature images at three different resolutions. This represents the input of the Kth MCA module. This represents the output of the Kth MCA module. The image feature pairs representing the output of each attention layer in the Kth MCA module are upsampled to restore the input resolution and then merged and compressed along the channels to obtain image feature pairs.

[0019] 3. In feature (2), optical flow prediction for consecutive frames, two consecutive frames of image I t and I t+1 After passing through an image feature enhancement network based on a multi-scale cross-attention module (MCA), image feature pairs F1 and F2 are finally obtained. Then, the correlation between the features of the two images is calculated to compare the feature similarity of each pixel in F1 with all pixels in F2. The operation is defined as follows:

[0020]

[0021] Here, CM represents the correlation matrix, where each element represents the correlation between coordinates in F1 and coordinates in F2; then, the last two dimensions of CM are normalized using a softmax operation to obtain the matching probability distribution M, which is defined as follows:

[0022]

[0023] Then, the correspondence matrix is ​​obtained by multiplying the weighted average of the 2D coordinates of the pixel grid G ​​with the matching probability distribution M. The size of the pixel grid G ​​is the same as that of F1 and F2, and its operation is defined as follows:

[0024]

[0025] Finally, the coordinate difference of the corresponding pixels is calculated to obtain the optical flow (OF), and the operation is as follows:

[0026]

[0027] 4. Pixel processing for occluded areas in feature (3): The high-quality optical flow estimation results of the matching area are propagated to the unmatched area through the self-similarity of the features. This is achieved through a self-attention layer, which is calculated using a sliding window method. Its definition is as follows:

[0028]

[0029] in This indicates the output optical flow before upsampling, and the final optical flow obtained by upsampling.

[0030] Compared with existing technologies, the advantages of this invention are as follows: This invention proposes a multi-scale cross-attention module (MCA), which uses the MCA module to complete the relevant information between different image patches in the same feature image, enabling the network to learn multi-scale image information. Simultaneously, it solves the pixel occlusion problem by modeling image self-similarity, thereby obtaining the final predicted optical flow. Attached Figure Description

[0031] Figure 1 This is a flowchart of the present invention;

[0032] Figure 2 This is a diagram of the optical flow network structure based on the multi-scale spatial cross-attention module (MCA).

[0033] Figure 3 To initialize the structure diagram;

[0034] Figure 4 This is a structural diagram of the Multi-Scale Spatial Cross-Attention Module (MCA).

[0035] Figure 5 Here is a diagram of the Concat structure in the MCA module;

[0036] Figure 6 This is a diagram of the cross-attention layer structure;

[0037] Figure 7 Diagram of the optical flow estimation module;

[0038] Figure 8 This is a structural diagram of the occlusion handling module. Detailed Implementation

[0039] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0040] Combination Figures 1-8 The present invention is achieved through the following steps:

[0041] Step 1: Construct an image feature enhancement network based on the multi-scale spatial cross-attention module (MCA).

[0042] like Figure 2 As shown, input two consecutive frames of images I t and I t+1The convolutional network for feature extraction consists of a 7×7 convolution with a stride of 2, a 1×1 convolution, and three residual modules, each composed of two 3×3 convolutions with a stride of 1. The feature extraction module then produces... The image feature pairs are then downsampled to H. l ×W l ×C, where H l With W l These are the height and width downsampled to the lowest resolution. These are manually specified constant values ​​that change depending on the resolution of the input image. They are then positionally encoded and input into a cross-attention layer, where the attention operation is as follows: Figure 6 As shown, it is defined as:

[0043]

[0044] Where CAtt represents cross-attention computation; Q, K, and V represent the linear projections of the input features, specifically Q from the source features F. s i Export, F s K and V represent the source features, i represents the feature index at different times; K and V represent the target features F. t i Export F t Represents the target features, where i represents the feature index at different times; T represents the matrix transpose operation, softmax(·) represents the normalization operation, and D represents the Q and K dimensions; W q W k W v Let M represent three distinct parameter matrices, whose global cross-attention scores are used to update the source features through the FFN layer. The update operation is defined as follows:

[0045]

[0046] Where Cat represents the Concat operation, then and The image is upsampled to restore the input resolution, and then fed into a multi-scale feature matching network composed of MCA modules. The entire multi-scale feature matching network contains N MCA modules. Each MCA module has three different resolution image features connected to a cross-attention layer, with the corresponding output attention score being M. c M m and M f Three different resolution image feature pairs are obtained by step average pooling downsampling, then upsampling to restore them to a uniform resolution after passing through an attention layer. These features are then merged along the channels and passed through a convolutional layer, finally added to the input image feature pairs for output. For the Kth MCA block, at a resolution of (H...l W l For the feature image pairs, pixel-wise cross-attention calculation is performed. For the other two resolutions of the image features, they are first divided, and the original feature F is then processed. S i The image feature blocks are divided into S×S patches, and a total of 10 patches are formed. Then input the cross-attention module, where M c M m and M f The calculation method is the same as the M calculation method described above. The features output from each attention layer are upsampled to restore the input resolution, and then merged and compressed along the channels to obtain... Then, after passing through a single-layer convolutional network, the output is processed as follows:

[0047]

[0048] Where LN represents the normalization layer, ConV represents the convolutional layer, and Cat represents the Concat operation; The output features of the cross-attention module represent feature images at three different resolutions. This represents the input of the Kth MCA module. This represents the output of the Kth MCA module. The image feature pairs representing the output of each attention layer in the Kth MCA module are upsampled to restore the input resolution and then merged and compressed along the channels to obtain image feature pairs.

[0049] Step 2: Construct an optical flow estimation network.

[0050] like Figure 7 As shown, two consecutive video images I t and I t+1 The input to the image feature enhancement network first obtains preliminary image features through a convolutional neural network. Since the two sets of features lack spatial location concepts, fixed two-dimensional sine and cosine positional codes are added to the features. Adding positional information allows the matching process to consider not only feature similarity but also their spatial distance, which helps resolve ambiguities and improve performance. After adding positional information, the features pass through N MCA modules, finally outputting image feature pairs F1 and F2. Considering that corresponding pixels in two frames should have high similarity, the correlation between the two image features is calculated to compare the feature similarity of each pixel in F1 relative to all pixels in F2. This step can be accomplished by a simple dot product, defined as follows:

[0051]

[0052] Here, CM represents the correlation matrix, where each element represents the correlation between coordinates in F1 and coordinates in F2. A simple method to determine the correspondence is to directly take the position with the highest correlation. However, this operation is non-differentiable, which hinders end-to-end training. To address this issue, a differentiable matching layer is used, which involves using a softmax operation to normalize the last two dimensions of CM to obtain the matching probability distribution M, defined as follows:

[0053]

[0054] Then, the correspondence matrix is ​​obtained by multiplying the weighted average of the 2D coordinates of the pixel grid G ​​with the matching probability distribution M. The size of the pixel grid G ​​is the same as that of F1 and F2, and its operation is defined as follows:

[0055]

[0056] Finally, the coordinate difference of the corresponding pixels is calculated to obtain the optical flow (OF), and the operation is as follows:

[0057]

[0058] Step 3: Process the pixels in the occluded area.

[0059] The principle behind the aforementioned optical flow prediction is that positional information is added to the features before they enter the correlation layer. First, significant feature points are found through autocorrelation. Then, the corresponding positions of feature points in two images are found through cross-attention. Clearly, the difference in positional information between the feature points in the two images is the optical flow. However, this process assumes that there are no occluded pixels. Once occluded pixels exist, the feature coordinates of the target image cannot correspond to the occluded pixel positions, resulting in inaccurate optical flow prediction. However, since the target image and the optical flow image have a certain degree of structural similarity, to resolve the ambiguity caused by occlusion, the idea is to allow the network to reason at a higher level. That is, after implicitly inferring which pixels are similar in appearance feature space, the motion features of similar pixels are globally aggregated. Assuming that by finding points with similar appearances in the reference frame, the network will be able to find points with similar motions. This is because the motion of points on a single object is usually uniform. This statistical bias is used to propagate motion information from high-confidence non-occluded pixels to low-confidence occluded pixels. Here, confidence can be interpreted as whether there is a clear match, i.e., a high correlation value at the correct displacement. Transformers networks are renowned for their ability to model long-term dependencies. In the self-attention mechanism of Transformers, the query, key, and value all originate from the same feature vector. Unlike the self-attention mechanism in Transformers, a generalized attention variant is used here. The query and key features are outputs of the image feature network, used to model appearance self-similarity in the first frame. The value features are optical flow projections passed through a softmax layer; the output optical flow itself is encoded with 4D correlated volume. The attention matrix computed based on the query and key features is used to aggregate the value features as a motion-hidden representation.

[0060] like Figure 8 As shown, high-quality optical flow estimation results from matching regions are propagated to unmatched regions through feature self-similarity. This method is implemented through a self-attention layer, which calculates attention using a sliding window approach, defined as follows:

[0061]

[0062] in This represents the output optical flow before upsampling. The final optical flow is obtained by upsampling.

[0063] Step 4: Input two consecutive frames of images into the network input and perform supervised training on the network using the overall network loss function. Use GroundTruth to monitor all traffic predictions.

[0064] Step 5: Input two consecutive frames of images into the trained model for testing, and output the corresponding estimated optical flow.

[0065] Two consecutive frames of images I t and I t+1 The input is first fed into the feature extraction module, then through a convolutional layer with a stride of 2 and a size of 7×7, followed by three consecutive residual blocks, each consisting of two convolutions with a stride of 1 and a size of 3×3. Finally, it passes through a 1×1 convolutional layer. The number of kernels in each convolutional layer is 64, 64, 96, 128, or 128. Except for the last convolutional layer, each convolutional layer is followed by a normalization layer and a ReLU activation layer. The result after passing through the feature extraction module is... The image feature pairs are then downsampled to H using straddle average pooling. l ×W l ×C, where H l With W l These are the height and width downsampled to the lowest resolution. They are manually specified constant values ​​that change depending on the resolution of the input image. Then, after positional encoding, they are input into a global cross-attention layer, which is defined as follows:

[0066]

[0067]

[0068] Where CAtt represents cross-attention computation; Q, K, and V represent the linear projections of the input features, specifically Q from the source features F. s i Export, F s K and V represent the source features, i represents the feature index at different times; K and V represent the target features. Export F t Represents the target features, where i represents the feature index at different times; T represents the matrix transpose operation, softmax(·) represents the normalization operation, and D represents the Q and K dimensions; W q W k W v Let M represent three distinct parameter matrices, whose global cross-attention scores are used to update the source features through the FFN layer. The update operation is defined as follows:

[0069]

[0070] Where Cat represents the Concat operation, then and The image is restored to its input resolution via bilinear upsampling, and then fed into a multi-scale feature matching network composed of MCA modules. The entire multi-scale feature matching network contains N MCA modules. Each MCA module contains three image feature pairs at different resolutions that pass through a cross-attention layer, with corresponding output attention scores of M. c M m and M f The image features at three different resolutions are obtained by step average pooling downsampling, then restored to a uniform resolution using bilinear regression after passing through an attention layer. They are then merged along the channels and passed through a convolutional layer, finally added to the input image feature pairs for output. For the Kth MCA block, at a resolution of (H... l W l For the feature image pairs, pixel-wise cross-attention calculation is performed. For the other two resolutions of the image features, they are first divided, and the original feature F is then processed. S i The image feature blocks are divided into S×S patches, and a total of 10 patches are formed. One patch. Then input the cross-attention module, where M... c M m and M f The calculation method is the same as the M calculation method described above. The features output from each attention layer are upsampled to restore the input resolution, and then merged and compressed along the channels to obtain... Then, after passing through a single-layer convolutional network, the output is processed as follows:

[0071]

[0072] Where LN represents the normalization layer, ConV represents the convolutional layer, and Cat represents the Concat operation; The output features of the cross-attention module represent feature images at three different resolutions. This represents the input of the Kth MCA module. This represents the output of the Kth MCA module. The image feature pairs output by each attention layer in the Kth MCA module are upsampled to restore the input resolution, merged along the channels, and then compressed to obtain image feature pairs. After passing through N MCA modules, the final image features are output, resulting in image feature pairs F1 and F2. The correlation between the two image features is then calculated to compare the feature similarity of each pixel in F1 relative to all pixels in F2. This step can be accomplished by a simple dot product, defined as follows:

[0073]

[0074] Here, CM represents the correlation matrix, where each element represents the correlation between coordinates in F1 and coordinates in F2. The matching probability distribution M is obtained by normalizing the last two dimensions of CM using the softmax operation, as defined below:

[0075]

[0076] Then, the correspondence matrix is ​​obtained by multiplying the weighted average of the 2D coordinates of the pixel grid G ​​with the matching probability distribution M. The size of the pixel grid G ​​is the same as that of F1 and F2, and its operation is defined as follows:

[0077]

[0078] Finally, the coordinate difference of the corresponding pixels is calculated to obtain the optical flow (OF), and the operation is as follows:

[0079]

[0080] Then, the high-quality optical flow estimation results of the matching region are propagated to the unmatched region through the self-similarity of features. This method is implemented through a self-attention layer, which is calculated using a sliding window approach, as defined below:

[0081]

[0082] in This represents the output optical flow before upsampling. The final optical flow is obtained by upsampling.

[0083] In summary, this invention provides an optical flow estimation method based on multi-scale global cross-matching, comprising the following steps: 1. Constructing an image feature enhancement network based on a multi-scale cross-attention module (MCA); 2. Constructing an optical flow estimation module; 3. Constructing an occlusion region pixel processing module; 4. Inputting two consecutive frames of images into the network input terminal for supervised training; 5. Inputting two consecutive frames of images into the trained model for testing, and outputting the corresponding estimated optical flow. This invention proposes a multi-scale cross-attention module (MCA), which utilizes the MCA module to complete the relevant information between different image patches in the same feature image, enabling the network to learn image information at multiple resolutions. Simultaneously, it solves the pixel occlusion problem by modeling image self-similarity, thereby obtaining the final predicted optical flow.

Claims

1. An optical flow estimation method based on multi-scale global cross-matching, characterized in that: (1) Construct an image feature enhancement network based on multi-scale cross attention module, including a feature extraction convolutional network and a multi-scale feature matching network composed of MCA modules. First, the input image is convolved to extract image features. Then, the feature image is positionally encoded and input into the multi-scale feature matching network composed of MCA modules. The entire multi-scale feature matching network contains N MCA modules. Each MCA module performs correlation calculation on three different resolutions of the input image. The three different resolution image pairs are obtained by downsampling, and after passing through the attention layer, they are restored to a unified resolution by upsampling and fused. Then, they are added to the input and output. (2) Construct an optical flow estimation module; input the two frames of image features output from the last layer of the multi-scale feature matching network into the optical flow estimation module for prediction. This module consists of a global matching module and a softmax layer; perform dot product operation on the input features to obtain global correlation, and then normalize the last two dimensions of global correlation through softmax to obtain the matching probability. Then, multiply the matching probability by the weighted average of the 2D coordinates of the pixel grid points to obtain the correspondence matrix; finally, the optical flow is obtained by calculating the coordinate difference between corresponding points. (3) Construct a pixel processing module for occlusion areas; this module consists of a self-attention layer, which inputs the 2D optical flow output by the optical flow estimation module and the target map features output by the multi-scale feature matching network into the self-attention layer, and then adds them to the 2D optical flow to obtain the final optical flow; (4) Input two consecutive frames of images into the network input terminal and perform supervised training; (5) Input two consecutive frames of images into the trained model for testing, and output the corresponding estimated optical flow.

2. The optical flow estimation method based on multi-scale global cross-matching according to claim 1, characterized in that: In step (1), two consecutive frames of images I t and I t+1 First, the input is obtained from a feature extraction network composed of convolutions. The feature map is defined as follows: H and W represent the height and width of the input image, respectively, and C represents the number of channels in the input image. The feature map is then downsampled to H. l ×W l ×C, where H l With W l These are the height and width downsampled to the lowest resolution. They are manually specified constant values ​​that change depending on the resolution of the input image. After positional encoding, they are input to a cross-attention layer, whose attention operation is defined as follows: Where CAtt represents cross-attention computation; Q, K, and V represent the linear projections of the input features, specifically Q from the source features F. s i Export, F s The source feature is represented by i, where i represents the feature index at different times. K,V from target features F t i Export F t represents the target feature, and i represents the feature index at different times; T represents the matrix transpose operation, softmax(·) represents the normalization operation, and D represents the Q and K dimensions; W q W k W v Representing three distinct parameter matrices, the global cross-attention score M is used to update the source features through the FFN layer of a feedforward neural network. The update operation is defined as follows: Where Cat represents the Concat operation, the updated output characteristics. and The image is then upsampled to the input size and then fed into a multi-scale image feature matching network consisting of MCA modules.

3. The optical flow estimation method based on multi-scale global cross-matching according to claim 2, characterized in that: In step (1), the MCA module obtains three different resolution image feature pairs through downsampling, which are H from low to high. l ×W l ×C、 as well as The three feature resolutions correspond to an attention score of M. c M m and M f , For the Kth MCA block, at a resolution of (H l W l For the feature image pairs, pixel-wise cross-attention calculation is performed. For the other two resolution features, they are first divided, and the original feature F is then processed. S i The image feature blocks are divided into S×S patches, and a total of 10 patches are formed. Each patch is then input into the cross-attention module; the features output from each attention layer are upsampled to restore the input size, and then merged and compressed along the channels to obtain the desired result. Then, after passing through a single-layer convolutional network, the output is processed as follows: Where LN represents the normalization layer, ConV represents the convolutional layer, and Cat represents the Concat operation; The output features of the cross-attention module represent feature images at three different resolutions. This represents the input of the Kth MCA module. This represents the output of the Kth MCA module. The image feature pairs representing the output of each attention layer in the Kth MCA module are upsampled to restore the input resolution and then merged and compressed along the channels to obtain image feature pairs.

4. The optical flow estimation method based on multi-scale global cross-matching according to claim 1, characterized in that: In step (2), optical flow prediction for consecutive frames is performed on two consecutive frames of image I. t and I t+1 After passing through an image feature enhancement network based on a multi-scale cross-attention module (MCA), image feature pairs F1 and F2 are finally obtained. Then, the correlation between the features of the two images is calculated to compare the feature similarity of each pixel in F1 with all pixels in F2. The operation is defined as follows: Here, CM represents the correlation matrix, where each element represents the correlation between coordinates in F1 and coordinates in F2; then, the last two dimensions of CM are normalized using a softmax operation to obtain the matching probability distribution M, which is defined as follows: Then, the correspondence matrix is ​​obtained by multiplying the weighted average of the 2D coordinates of the pixel grid G ​​with the matching probability distribution M. The size of the pixel grid G ​​is the same as that of F1 and F2, and its operation is defined as follows: Finally, the coordinate difference of the corresponding pixels is calculated to obtain the optical flow (OF), and the operation is as follows:

5. The optical flow estimation method based on multi-scale global cross-matching according to claim 1, characterized in that: The pixel processing of the occluded area in step (3); The high-quality optical flow estimation results of the matching region are propagated to the unmatched region by leveraging the self-similarity of features. This is achieved through a self-attention layer, which is calculated using a sliding window approach. The self-attention layer is defined as follows: in This indicates the output optical flow before upsampling, and the final optical flow obtained by upsampling.

Images