A multi-scale optical flow estimation method based on scale experts

Abstract

This invention aims to provide a multi-scale optical flow estimation method based on scale experts, comprising the following steps: S1, inputting two temporally adjacent images into a ResNet-18 feature extraction network to extract features from the first and second frames respectively; S2, initializing optical flow and hidden features to obtain initial optical flow features and initial hidden features, both of which are all zero tensors; S3, constructing a coarse optical flow update module A to process the optical flow features and hidden features of the two frames; S4, constructing a coarse optical flow update module B to process the optical flow features and hidden features of the two frames; S5, constructing a coarse optical flow update module C to process the optical flow features and hidden features of the two frames; S6, constructing a fine optical flow update module to process the optical flow features and hidden features of the two frames; S7, upsampling the optical flow features to obtain the final optical flow features at the original resolution. This invention can achieve an effective balance between optical flow estimation accuracy and inference efficiency.

Description

Technical Field

[0001] This invention relates to the field of computer image processing, and more specifically to a multi-scale optical flow estimation method based on scale experts. Background Technology

[0002] In recent years, deep learning models, represented by RAFT, have significantly improved the performance of optical flow estimation by constructing a correlation cost volume between the source and target frames and using recurrent units for iterative optimization. To capture fine-grained motion and enhance robustness to large displacements, subsequent research has increasingly adopted a "coarse-to-fine" multi-scale strategy. This strategy typically involves a U-Net-like architecture, using pyramid features to progressively refine the optical flow field. For example, methods such as MS-RAFT+ and CCMR have demonstrated that multi-scale designs can generate more detailed flow fields; other works, such as DPFlow, attempt to achieve a balance between efficiency and accuracy by optimizing the pyramid structure.

[0003] Despite the aforementioned progress, a significant and often overlooked architectural flaw remains in existing multi-scale optical flow estimation methods: the common practice of using a single optical flow update module with shared weights across all resolution levels. This "one-size-fits-all" design forces a fixed module to process feature inputs from drastically different resolutions with varying statistical properties, leading to two main technical bottlenecks: 1. High computational cost due to forced feature alignment: In order for the shared updater to work, the encoder is forced to compress or stretch features from each scale (from 1 / 4 to 1 / 32) into a single, uniform feature space. This requirement necessitates extremely expensive convolutional computations when processing high-resolution features, resulting in high memory usage and slow inference speed, making it difficult to meet the needs of real-time applications.

[0004] 2. Impaired representational ability due to feature consistency: At low-resolution levels, rich global contextual information should be included to handle large displacements and occlusions. However, to maintain format consistency with high-resolution features, this valuable global information is often sacrificed or compressed. This results in poor model performance in scenarios requiring strong global inference, with the update module receiving insufficient global information.

[0005] Therefore, existing optical flow estimation methods face a significant trade-off: high-resolution refinement comes with prohibitively high computational costs, while forced feature unification weakens the model's expressive power. How to design a method that can perform targeted processing based on characteristics at different scales, reducing the computational load at high resolution while preserving global inference capabilities at low resolution, is a pressing problem that current optical flow estimation techniques need to solve. Summary of the Invention

[0006] This invention aims to provide a multi-scale optical flow estimation method based on scale experts. This method is designed based on the differences in statistical characteristics of features at different resolution levels, breaking the limitation of cross-scale weight sharing in traditional multi-scale methods. By constructing a scale-specific expert group, it captures large-scale motion and global context information using a global correlation cost volume at low resolution levels, and abandons time-consuming cost volume calculations at high resolution levels, using only distortion features and attention mechanisms to correct local minor details. This eliminates computational redundancy at high resolution and preserves semantic representation at low resolution, thereby achieving an effective balance between optical flow estimation accuracy and inference efficiency, and is applicable to various motion-related detection tasks.

[0007] The technical solution of the present invention is as follows: The multi-scale optical flow estimation method based on scale experts includes the following steps: S1. Input two temporally adjacent images into the ResNet-18 feature extraction network to extract feature sets for the first and second frames respectively. The feature sets of the first and second frames contain low-resolution feature maps with resolutions reduced to 1 / 32, 1 / 16, and 1 / 8 of the original image, and high-resolution feature maps with resolutions reduced to 1 / 4 of the original image. Take the low-resolution (1 / 32, 1 / 16, 1 / 8) and high-resolution feature maps as four context feature maps. The context feature maps of the first frame are as follows: The context feature maps of the second frame image are as follows: We take the low-resolution feature maps of 1 / 32, 1 / 16, and 1 / 8 as matching features. The matching features of the first frame image are as follows: The matching features of the second frame image are as follows: ; S2. Initialize optical flow and hidden features to obtain initial optical flow features and initial hidden features, both of which are all zero tensors; S3. Construct coarse optical flow update module A, inputting initial optical flow features and initial hidden features, and simultaneously inputting matching features. , and context features , The optical flow features of the two frames were obtained by processing them separately. and hidden features ; S4. Construct coarse optical flow update module B, inputting optical flow features. and hidden features Simultaneously input matching features , and context features , The optical flow features of the two frames of images are obtained by inputting them separately and processing them separately. and hidden features ; S5. Construct the coarse optical flow update module C, and input the optical flow features. and hidden features Simultaneously input matching features , and context features , The optical flow features of the two frames of images are obtained by inputting them separately and processing them separately. and hidden features ; S6. Construct a fine optical flow update module and input optical flow features. and hidden features Simultaneously input contextual features , The optical flow features of the two frames of images are obtained by inputting them separately and processing them separately. and hidden features ; S7. Use the convex upsampling algorithm to analyze optical flow characteristics. Upsampling is performed to obtain the final optical flow features at the original resolution. .

[0008] Step S1 specifically includes the following steps: S11, Transfer the first frame image Second frame image Using bilinear interpolation algorithm, samples were downsampled to different resolutions. Four downsampled images of different resolutions were obtained from the two frames. and ,in , , , ; for downsampled images and Perform 1×1 convolutions separately to obtain edge features. , ; S12, Transfer the first frame image Second frame image The images are processed sequentially through a 3×3 convolution with a stride of 2, a LayerNorm layer, another 3×3 convolution with a stride of 2, and a LayerNorm layer, resulting in intermediate results A1 and A2 where the resolution of the two frames is reduced to 1 / 4 of the original image. Compare the intermediate result A1 with the edge features of the same resolution. After addition, the results are processed by two convolutional modules to obtain the high-resolution contextual features of the first frame image. ; Compare the intermediate result A2 with the edge features of the same resolution. After addition, the results are processed by two convolutional modules to obtain the high-resolution contextual features of the first frame image. ; The convolutional module contains, in sequence, a 3×3 convolutional layer with a stride of 1, a ReLU activation function, a LayerNorm layer, and another 3×3 convolutional layer with a stride of 1. S13, High-resolution contextual features , After being processed by 2×2 convolution downsampling with a stride of 2, intermediate results B1 and B2 with a resolution reduced to 1 / 8 are obtained. Compare the intermediate result B1 with the edge features of the same resolution. After addition, the results are processed sequentially through two POLA-Transformer modules. The results are divided into two paths, each of which undergoes a 1×1 convolution with a stride of 1 to obtain matching features. and context features ; Compare the intermediate result B2 with the edge features of the same resolution. After addition, the results are processed sequentially through two POLA-Transformer modules. The results are divided into two paths, each of which undergoes a 1×1 convolution with a stride of 1 to obtain matching features. and context features ; S14. Matching features and context features The features are concatenated, and the results are then subjected to 2×2 convolution downsampling with a stride of 2 to obtain an intermediate result C1 with a resolution of 1 / 16. The intermediate result C1 is then compared with edge features of the same resolution. After addition, the results are processed sequentially through two POLA-Transformer modules. The results are divided into two paths, each of which undergoes a 1×1 convolution with a stride of 1 to obtain matching features. and context features ; Matching features and context features The feature dimensions are concatenated, and the results are then processed by 2×2 convolution downsampling with a stride of 2 to obtain an intermediate result C2 with a resolution of 1 / 16. Compare the intermediate result C2 with edge features of the same resolution. After addition, the results are processed sequentially through two POLA-Transformer modules. The results are divided into two paths, each of which undergoes a 1×1 convolution with a stride of 1 to obtain matching features. and context features ; S15. Matching features and context features The features are concatenated, and the results are then subjected to 2×2 convolution downsampling with a stride of 2 to obtain an intermediate result D1 with a resolution of 1 / 16. The intermediate result D1 is then compared with edge features of the same resolution. After addition, the results are processed sequentially through two POLA-Transformer modules. The results are divided into two paths, each of which undergoes a 1×1 convolution with a stride of 1 to obtain matching features. and context features ; Matching features and context features The feature dimensions are concatenated, and the results are then processed by 2×2 convolution downsampling with a stride of 2 to obtain an intermediate result D2 with a resolution of 1 / 16. Compare the intermediate result D2 with the edge features of the same resolution. After addition, the results are processed sequentially through two POLA-Transformer modules. The results are divided into two paths, each of which undergoes a 1×1 convolution with a stride of 1 to obtain matching features. and context features .

[0009] Matching features and context features The feature dimensions are concatenated, and the results are then processed by 2×2 convolution downsampling with a stride of 2 to obtain an intermediate result D2 with a resolution of 1 / 16. Compare the intermediate result D2 with the edge features of the same resolution. After addition, the results are processed sequentially through two POLA-Transformer modules. The results are divided into two paths, each of which undergoes a 1×1 convolution with a stride of 1 to obtain matching features. and context features .

[0010] The process of initializing optical flow and hidden features in step S2 is as follows: S21. Set the initial optical flow feature to a zero tensor, with a spatial resolution of . The channel count is 2, used to represent the initial displacement in the horizontal and vertical directions; S22. Set the initial hidden features to all-zero tensors, with a spatial resolution consistent with the initial optical flow field, i.e. The number of channels is D, which is used to store and transmit global context information and motion history information in subsequent loop iterations.

[0011] In step S3, the coarse optical flow update module A operates as follows: S31. Calculate the matching features of the first frame. Second frame matching features The dot product is calculated using the following formula: ,in The coordinates of the matching features in the first and second frames are respectively. ; The correlation of global feature similarity at a given resolution; S32. Based on the initial optical flow characteristics and global correlation characteristics Perform standard rigid body sampling to obtain local features of a 4×4 area in the optical flow pointing coordinates. Correlation; based on initial optical flow features and contextual features of the second frame. Perform standard rigid body sampling to obtain the context sampling features from the second frame to the first frame. ; S33. Correlation of local features The initial optical flow features are input into a standard motion encoder constructed using the RAFT model to extract motion features containing motion information. ; motion characteristics The features are concatenated with the initial hidden features along the feature dimension and then input into the ConvNeXt module for processing to obtain the non-occluded features. Then, the context sampling features from the second frame to the first frame are used. First frame context features and non-occlusion features The images are mapped to vectors using standard image embedding operations, then input into the POLA-Transformer module to capture global long-range dependencies and occlusion relationships, and output occlusion features. ; S34. Non-occluded features and The occlusion features are concatenated and fused through a 1×1 convolutional layer to generate updated hidden features. ; Hide features The new optical flow features are estimated by inputting them into the standard prediction head constructed from the RAFT model. ; S35. Repeat steps S32 to S34 once to obtain the result. The hidden features and optical flow features at a given resolution are collectively referred to as optical flow features. and hidden features .

[0012] In step S4, the coarse optical flow update module B processes the process as follows: S41, Optical flow characteristics and hidden features Its spatial resolution is magnified by a factor of 2 using bilinear interpolation, so that its resolution becomes This yields the initial optical flow feature F2' and the initial hidden feature H2' of the current level; S42. Calculate matching features and matching features The dot product is calculated using the following formula: ,in These are the coordinates of the matched features in the first and second frames, respectively. Recorded at resolution The correlation of global feature similarity; S43. Based on the initial optical flow characteristics F2' and global correlation characteristics Perform standard rigid body sampling to obtain the local feature correlation of a local 4×4 area in the optical flow pointing coordinates. Based on the initial optical flow feature F2' and the context features of the second frame. Perform standard rigid body sampling to obtain the context sampling features from the second frame to the first frame. ; S44. The local feature correlation obtained in step S43 The initial optical flow feature F2' is input into a standard motion encoder constructed using the RAFT model to extract motion features containing motion information. ; motion characteristics The feature is concatenated with the initial hidden feature H2' along the feature dimension and then fed into a standard ConvNeXt module to capture motion information of the local unoccluded region, outputting the unoccluded feature. ; Context sampling features from the second frame to the first frame Contextual features and non-occlusion features The images are mapped to vectors using standard image embedding operations and then directly input into the POLA-Transformer module to capture global long-range dependencies and occlusion relationships, outputting occlusion features. ; S45, Non-occluded features With occlusion features The features are concatenated and then fused using a 1×1 convolutional layer to generate updated hidden features. ; Hide features The new optical flow value is estimated by inputting it into the standard prediction head constructed by the RAFT model. ; S46. Repeat steps S43 to S45 once to obtain the result. The hidden features and optical flow features at a given resolution are collectively referred to as optical flow features. and hidden features .

[0013] In step S5, the coarse optical flow update module C processes the process as follows: S51, Optical flow characteristics and hidden features Its spatial resolution is magnified by a factor of 2 using bilinear interpolation algorithm, thus increasing its resolution size. This yields the initial optical flow feature F3' and the initial hidden feature H3' of the current level; S52. Calculate matching features and matching features The dot product is calculated using the following formula: ,in These are the coordinates of the matched features in the first and second frames, respectively. Recorded in Correlation of global feature similarity at different resolutions; S53. Based on the initial optical flow characteristics F3' and global correlation characteristics Perform standard rigid body sampling to obtain the local feature correlation of a local 4×4 area in the optical flow pointing coordinates. Based on the initial optical flow feature F3' and the context features of the second frame. Perform standard rigid body sampling to obtain the context sampling features from the second frame to the first frame. ; S54. The local feature correlation obtained in step S53 The initial optical flow feature F3' is input into a standard motion encoder constructed using the RAFT model to extract motion features containing motion information. ; motion characteristics The feature is concatenated with the initial hidden feature H3' along the feature dimension and then fed into a standard ConvNeXt module to capture motion information of the local unoccluded region, outputting the unoccluded feature. ; Context sampling features from the second frame to the first frame First frame context features and non-occlusion features The images are mapped to vectors using standard image embedding operations and then directly input into the POLA-Transformer module to capture global long-range dependencies and occlusion relationships, outputting occlusion features. ; S55, Non-occlusion features With occlusion features The features are concatenated and then fused using a 1×1 convolutional layer to generate updated hidden features. ; Hide features The new optical flow value is estimated by inputting it into the standard prediction head constructed by the RAFT model. ; S56. Repeat steps S53 to S55 once to obtain the result. The hidden features and optical flow features at a given resolution are collectively referred to as optical flow features. and hidden features .

[0014] In step S6, the fine optical flow update module processes the process as follows: SS61, Optical flow characteristics and hidden features Its spatial resolution is magnified by a factor of 2 using bilinear interpolation algorithm, thus increasing its resolution size. This yields the initial optical flow F4' and initial hidden feature H4' for the current level; S62. At this high-resolution level, the construction of the global correlation cost volume and correlation sampling operations are not performed; instead, fine-grained updates are directly implemented. Standard rigid body sampling is performed based on the initial optical flow F4' and the context features of the second frame to obtain the context sampling features from the second frame to the first frame. ; S63. The context sampling features obtained in step S62 High-resolution contextual features of the first frame The initial optical flow F4' is concatenated with the vector, mapped to a vector using standard image embedding operations, and directly input into the POLA-Transformer module to capture global long-range dependencies and occlusion relationships, outputting the updated hidden features, which are the hidden features. ; Hide features The new optical flow features are estimated by inputting them into the standard prediction head constructed by the RAFT model; these optical flow features are then called optical flow characteristics. .

[0015] The specific operation of step S7 is as follows: S71, Hide Features After sequential processing with 3×3 convolution, ReLU activation function, and 3×3 convolution, a 4x upsampling mask is output. This 4x upsampling mask and the optical flow features are then upsampled by 4x using a standard convex upsampling algorithm to obtain the optical flow features at the original resolution. .

[0016] The beneficial effects of this invention are as follows: The method of this invention first utilizes the ResNet-18 feature extraction network with shared weights to extract feature pyramids at multiple resolution levels and construct scale-specific expert groups. At low-resolution levels, a global correlation cost volume is constructed using matching features, and a motion encoder and a two-branch coarse-grained updater are combined to capture large-amplitude motion and global context information. During inter-level transfer, guided optical flow and shared hidden states are used to achieve cross-scale information interaction. At high-resolution levels, the computationally intensive cost volume construction and sampling steps are abandoned, and only feature sampling and a single-branch fine-grained updater are used to correct small optical flow residuals. Finally, the original resolution optical flow field is recovered through convex upsampling.

[0017] The method of this invention designs differentiated processing strategies based on the statistical characteristics of features at different resolution levels, breaking the architectural limitation of weight sharing in traditional multi-scale optical flow estimation. While significantly reducing the computational cost and memory usage at high-resolution levels, it retains the global inference capability of low-resolution levels for large displacement and occlusion regions. This effectively solves the contradiction between high accuracy and high efficiency in existing technologies, and improves the robustness and accuracy of optical flow estimation in complex dynamic scenes.

[0018] The method of this invention is suitable for various motion-related detection tasks, including visual SLAM, video frame interpolation, target tracking, etc. Attached Figure Description

[0019] Figure 1 This is a flowchart illustrating the principle of the method of the present invention. Detailed Implementation

[0020] The present invention will now be described in detail with reference to the accompanying drawings and embodiments. Example 1

[0021] The multi-scale optical flow estimation method based on scale experts includes the following steps: S1. Input two temporally adjacent images into the ResNet-18 feature extraction network to extract feature sets for the first and second frames respectively. The feature sets of the first and second frames contain low-resolution feature maps with resolutions reduced to 1 / 32, 1 / 16, and 1 / 8 of the original image, and high-resolution feature maps with resolutions reduced to 1 / 4 of the original image. Take the low-resolution (1 / 32, 1 / 16, 1 / 8) and high-resolution feature maps as four context feature maps. The context feature maps of the first frame are as follows: The context feature maps of the second frame image are as follows: We take the low-resolution feature maps of 1 / 32, 1 / 16, and 1 / 8 as matching features. The matching features of the first frame image are as follows: The matching features of the second frame image are as follows: ; Specifically, the steps include the following: S11, Transfer the first frame image Second frame image Using bilinear interpolation algorithm, samples were downsampled to different resolutions. Four downsampled images of different resolutions were obtained from the two frames. and ,in , , , ; for downsampled images and Perform 1×1 convolutions separately to obtain edge features. , ; S12, Transfer the first frame image Second frame image The images are processed sequentially through a 3×3 convolution with a stride of 2, a LayerNorm layer, another 3×3 convolution with a stride of 2, and a LayerNorm layer, resulting in intermediate results A1 and A2 where the resolution of the two frames is reduced to 1 / 4 of the original image. Compare the intermediate result A1 with the edge features of the same resolution. After addition, the results are processed by two convolutional modules to obtain the high-resolution contextual features of the first frame image. ; Compare the intermediate result A2 with the edge features of the same resolution. After addition, the results are processed by two convolutional modules to obtain the high-resolution contextual features of the first frame image. ; The convolutional module contains, in sequence, a 3×3 convolutional layer with a stride of 1, a ReLU activation function, a LayerNorm layer, and another 3×3 convolutional layer with a stride of 1. S13, High-resolution contextual features , After being processed by 2×2 convolution downsampling with a stride of 2, intermediate results B1 and B2 with a resolution reduced to 1 / 8 are obtained. Compare the intermediate result B1 with the edge features of the same resolution. After addition, the results are processed sequentially through two POLA-Transformer modules. The results are divided into two paths, each of which undergoes a 1×1 convolution with a stride of 1 to obtain matching features. and context features ; Compare the intermediate result B2 with the edge features of the same resolution. After addition, the results are processed sequentially through two POLA-Transformer modules. The results are divided into two paths, each of which undergoes a 1×1 convolution with a stride of 1 to obtain matching features. and context features ; S14. Matching features and context features The features are concatenated, and the results are then subjected to 2×2 convolution downsampling with a stride of 2 to obtain an intermediate result C1 with a resolution of 1 / 16. The intermediate result C1 is then compared with edge features of the same resolution. After addition, the results are processed sequentially through two POLA-Transformer modules. The results are divided into two paths, each of which undergoes a 1×1 convolution with a stride of 1 to obtain matching features. and context features ; Matching features and context features The feature dimensions are concatenated, and the results are then processed by 2×2 convolution downsampling with a stride of 2 to obtain an intermediate result C2 with a resolution of 1 / 16. Compare the intermediate result C2 with edge features of the same resolution. After addition, the results are processed sequentially through two POLA-Transformer modules. The results are divided into two paths, each of which undergoes a 1×1 convolution with a stride of 1 to obtain matching features. and context features ; S15. Matching features and context features The features are concatenated, and the results are then subjected to 2×2 convolution downsampling with a stride of 2 to obtain an intermediate result D1 with a resolution of 1 / 16. The intermediate result D1 is then compared with edge features of the same resolution. After addition, the results are processed sequentially through two POLA-Transformer modules. The results are divided into two paths, each of which undergoes a 1×1 convolution with a stride of 1 to obtain matching features. and context features ; Matching features and context features The feature dimensions are concatenated, and the results are then processed by 2×2 convolution downsampling with a stride of 2 to obtain an intermediate result D2 with a resolution of 1 / 16. Compare the intermediate result D2 with the edge features of the same resolution. After addition, the results are processed sequentially through two POLA-Transformer modules. The results are divided into two paths, each of which undergoes a 1×1 convolution with a stride of 1 to obtain matching features. and context features .

[0022] S2. Initialize optical flow and hidden features to obtain initial optical flow features and initial hidden features, both of which are all zero tensors; The process of initializing optical flow and hidden features is as follows: S21. Set the initial optical flow feature to a zero tensor, with a spatial resolution of . The channel count is 2, used to represent the initial displacement in the horizontal and vertical directions; S22. Set the initial hidden features to all-zero tensors, with a spatial resolution consistent with the initial optical flow field, i.e. The number of channels is D, which is used to store and transmit global context information and motion history information in subsequent loop iterations.

[0023] S3. Construct coarse optical flow update module A, inputting initial optical flow features and initial hidden features, and simultaneously inputting matching features. , and context features , The optical flow features of the two frames were obtained by processing them separately. and hidden features ; The working process of coarse optical flow update module A is as follows: S31. Calculate the matching features of the first frame. Second frame matching features The dot product is calculated using the following formula: ,in The coordinates of the matching features in the first and second frames are respectively. ; The correlation of global feature similarity at a given resolution; S32. Based on the initial optical flow characteristics and global correlation characteristics Perform standard rigid body sampling to obtain local features of a 4×4 area in the optical flow pointing coordinates. Correlation; based on initial optical flow features and contextual features of the second frame. Perform standard rigid body sampling to obtain the context sampling features from the second frame to the first frame. ; S33. Correlation of local features The initial optical flow features are input into a standard motion encoder constructed using the RAFT model to extract motion features containing motion information. ; motion characteristics The features are concatenated with the initial hidden features along the feature dimension and then input into the ConvNeXt module for processing to obtain the non-occluded features. Then, the context sampling features from the second frame to the first frame are used. First frame context features and non-occlusion features The images are mapped to vectors using standard image embedding operations, then input into the POLA-Transformer module to capture global long-range dependencies and occlusion relationships, and output occlusion features. ; S34. Non-occluded features and The occlusion features are concatenated and fused through a 1×1 convolutional layer to generate updated hidden features. ; Hide features The new optical flow features are estimated by inputting them into the standard prediction head constructed from the RAFT model. ; S35. Repeat steps S32 to S34 once to obtain the result. The hidden features and optical flow features at a given resolution are collectively referred to as optical flow features. and hidden features .

[0024] S4. Construct coarse optical flow update module B, inputting optical flow features. and hidden features Simultaneously input matching features , and context features , The optical flow features of the two frames of images are obtained by inputting them separately and processing them separately. and hidden features ; S4. Construct coarse optical flow update module B, inputting optical flow features. and hidden features Simultaneously input matching features , and context features , The optical flow features of the two frames of images are obtained by inputting them separately and processing them separately. and hidden features ; S41, Optical flow characteristics and hidden features Its spatial resolution is magnified by a factor of 2 using bilinear interpolation, so that its resolution becomes This yields the initial optical flow feature F2' and the initial hidden feature H2' of the current level; S42. Calculate matching features and matching features The dot product is calculated using the following formula: ,in These are the coordinates of the matched features in the first and second frames, respectively. Recorded at resolution The correlation of global feature similarity; S43. Based on the initial optical flow characteristics F2' and global correlation characteristics Perform standard rigid body sampling to obtain the local feature correlation of a local 4×4 area in the optical flow pointing coordinates. Based on the initial optical flow feature F2' and the context features of the second frame. Perform standard rigid body sampling to obtain the context sampling features from the second frame to the first frame. ; S44. The local feature correlation obtained in step S43 The initial optical flow feature F2' is input into a standard motion encoder constructed using the RAFT model to extract motion features containing motion information. ; motion characteristics The feature is concatenated with the initial hidden feature H2' along the feature dimension and then fed into a standard ConvNeXt module to capture motion information of the local unoccluded region, outputting the unoccluded feature. ; Context sampling features from the second frame to the first frame Contextual features and non-occlusion features The images are mapped to vectors using standard image embedding operations and then directly input into the POLA-Transformer module to capture global long-range dependencies and occlusion relationships, outputting occlusion features. ; S45, Non-occluded features With occlusion features The features are concatenated and then fused using a 1×1 convolutional layer to generate updated hidden features. ; Hide features The new optical flow value is estimated by inputting it into the standard prediction head constructed by the RAFT model. ; S46. Repeat steps S43 to S45 once to obtain the result. The hidden features and optical flow features at a given resolution are collectively referred to as optical flow features. and hidden features .

[0025] S5. Construct the coarse optical flow update module C, and input the optical flow features. and hidden features Simultaneously input matching features , and context features , The optical flow features of the two frames of images are obtained by inputting them separately and processing them separately. and hidden features ; S51, Optical flow characteristics and hidden features Its spatial resolution is magnified by a factor of 2 using bilinear interpolation algorithm, thus increasing its resolution size. This yields the initial optical flow feature F3' and the initial hidden feature H3' of the current level; S52. Calculate matching features and matching features The dot product is calculated using the following formula: ,in These are the coordinates of the matched features in the first and second frames, respectively. Recorded in Correlation of global feature similarity at different resolutions; S53. Based on the initial optical flow characteristics F3' and global correlation characteristics Perform standard rigid body sampling to obtain the local feature correlation of a local 4×4 area in the optical flow pointing coordinates. Based on the initial optical flow feature F3' and the context features of the second frame. Perform standard rigid body sampling to obtain the context sampling features from the second frame to the first frame. ; S54. The local feature correlation obtained in step S53 The initial optical flow feature F3' is input into a standard motion encoder constructed using the RAFT model to extract motion features containing motion information. ; motion characteristics The feature is concatenated with the initial hidden feature H3' along the feature dimension and then fed into a standard ConvNeXt module to capture motion information of the local unoccluded region, outputting the unoccluded feature. ; Context sampling features from the second frame to the first frame First frame context features and non-occlusion features The images are mapped to vectors using standard image embedding operations and then directly input into the POLA-Transformer module to capture global long-range dependencies and occlusion relationships, outputting occlusion features. ; S55, Non-occlusion features With occlusion features The features are concatenated and then fused using a 1×1 convolutional layer to generate updated hidden features. ; Hide features The new optical flow value is estimated by inputting it into the standard prediction head constructed by the RAFT model. ; S56. Repeat steps S53 to S55 once to obtain the result. The hidden features and optical flow features at a given resolution are collectively referred to as optical flow features. and hidden features .

[0026] S6. Construct a fine optical flow update module and input optical flow features. and hidden features Simultaneously input contextual features , The optical flow features of the two frames of images are obtained by inputting them separately and processing them separately. and hidden features ; The fine optical flow update module processes the following steps: S61, Optical flow characteristics and hidden features Its spatial resolution is magnified by a factor of 2 using bilinear interpolation algorithm, thus increasing its resolution size. This yields the initial optical flow F4' and initial hidden feature H4' for the current level; S62. At this high-resolution level, the construction of the global correlation cost volume and correlation sampling operations are not performed; instead, fine-grained updates are directly implemented. Standard rigid body sampling is performed based on the initial optical flow F4' and the context features of the second frame to obtain the context sampling features from the second frame to the first frame. ; S63. The context sampling features obtained in step S62 High-resolution contextual features of the first frame The initial optical flow F4' is concatenated with the vector, mapped to a vector using standard image embedding operations, and directly input into the POLA-Transformer module to capture global long-range dependencies and occlusion relationships, outputting the updated hidden features, which are the hidden features. ; Hide features The new optical flow features are estimated by inputting them into the standard prediction head constructed by the RAFT model; these optical flow features are then called optical flow characteristics. .

[0027] S7. Use the convex upsampling algorithm to analyze optical flow characteristics. Upsampling is performed to obtain the final optical flow features at the original resolution. ;

[0028] The specific steps are as follows: S71, Hide Features After sequential processing with 3×3 convolution, ReLU activation function, and 3×3 convolution, a 4x upsampling mask is output. This 4x upsampling mask and the optical flow features are then upsampled by 4x using a standard convex upsampling algorithm to obtain the optical flow features at the original resolution. ; S8. Training Process: Construct a multi-scale supervised loss function to incorporate optical flow features. , , Upsampled to the original resolution and combined with optical flow features The L1-loss is calculated together with the standard optical flow labels, and the model is trained and optimized. The specific process of step S8 is as follows: S81. Using bilinear interpolation algorithm, optical flow is... Upsampled to the original image resolution Ensure that all predicted optical flow fields have the same spatial dimension as the standard optical flow labels; S82, Optical flow Upsampled images and final optical flow Standard optical flow labels are used to construct the L1-Loss loss function, and the calculation formula is as follows:

[0029] in, This is the actual optical flow value. These are the output optical flows at different resolutions; here we set... γ=0.8 ; S83. The AdamW optimizer is used to update the network parameters. The total batch size is set to 16. In the first stage, basic training is performed on the FlyingChairs dataset for 100,000 iterations. The initial learning rate is set to... Phase 2 then involves advanced training on the FlyingThings3D dataset for 200,000 iterations, with the learning rate adjusted to... Next, a mixed dataset containing samples from FlyingThings3D, KITTI2015, HD1K, and Sintel was constructed. In Phase 3, the dataset was fine-tuned 120,000 times, with the learning rate maintained at a constant value. Phase 4a fine-tuned the model on the KITTI2015 dataset, with a total of 2,000 iterations. Phase 4b was based on the model from Phase 3 and fine-tuned using Spring training samples, with a total of 120,000 iterations.

[0030] Example 2 Based on the model trained in Example 1, the method in Example 1 is used to test the existing dataset. The testing process is as follows: The validation data includes the Clean and Final branches of the Sintel dataset. In the Sintel validation, the final optical flow is calculated. and tags The mean endpoint error (EPE) is calculated using the following formula: .

[0031] Among them, optical flow and its labels are The data is as follows: H and W represent the height and width of the image, 2 represents the number of channels, and represent the offset in the x-direction and the offset in the y-direction, respectively.

[0032] In the KITTI dataset, ALL represents the average endpoint error of all images, and Non-occ represents the average endpoint error of the non-occluded region.

[0033] Inference Cost is the cost of inference. We tested it on an input image with a resolution of 960*640, using an RTX 4090. MACs(G) represent the number of multiply-accumulate operations, where G is the order of magnitude. Latency (ms) is the inference latency, measured in milliseconds (ms), representing the total time taken for the entire model to run.

[0034] The test results are shown in Table 1 below: .

[0035] As shown in Table 1, experimental results on benchmark datasets such as FlyingChairs, Sintel, and KITTI demonstrate that the method of this invention significantly optimizes computational efficiency while maintaining competitive optical flow estimation accuracy.

[0036] Experimental data show that the present invention achieves industry-leading endpoint error and outlier ratio on the KITTI dataset, verifying the model's robustness in complex scenarios. Crucially, by abandoning expensive cost volume construction at high-resolution levels and employing scale-specific expert design, the present invention significantly reduces floating-point operations compared to similar multi-scale methods such as MS-RAFT+, achieving an inference latency of only 38ms on RTX 4090 hardware, demonstrating state-of-the-art runtime efficiency.

[0037] Furthermore, qualitative analysis shows that the method of the present invention can effectively preserve high-frequency motion details, reduce the blurring and spillover effects of the optical flow field at the boundary of object motion, and improve the ability to resolve subtle motions and large displacements.

[0038] In summary, this invention successfully solves the problem of balancing computational cost and estimation accuracy in traditional methods, achieving the best balance between efficiency and accuracy in optical flow estimation.

[0039] Those skilled in the art will recognize that the embodiments described herein are intended to help the reader understand the principles of the invention, and should be understood that the scope of protection of the invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific modifications and combinations based on the technical teachings disclosed in this invention without departing from the spirit of the invention, and these modifications and combinations are still within the scope of protection of this invention.

Claims

1. A multi-scale optical flow estimation method based on scale experts, characterized in that, Includes the following steps: S1. Input two temporally adjacent images into the ResNet-18 feature extraction network to extract feature sets for the first and second frames respectively. These feature sets contain low-resolution feature maps with resolutions reduced to 1 / 32, 1 / 16, and 1 / 8 of the original image, and high-resolution feature maps with resolutions reduced to 1 / 4 of the original image. The low-resolution, 1 / 16, and 1 / 8 feature maps and the high-resolution feature map are taken as four context feature maps. The context feature maps for the first frame are as follows: The context feature maps of the second frame image are as follows: We take the low-resolution feature maps of 1 / 32, 1 / 16, and 1 / 8 as matching features. The matching features of the first frame image are as follows: The matching features of the second frame image are as follows: ; S2. Initialize optical flow and hidden features to obtain initial optical flow features and initial hidden features, both of which are all zero tensors; S3. Construct coarse optical flow update module A, inputting initial optical flow features and initial hidden features, and simultaneously inputting matching features. , and context features , The optical flow features of the two frames were obtained by processing them separately. and hidden features ; S4. Construct coarse optical flow update module B, inputting optical flow features. and hidden features Simultaneously input matching features , and context features , The optical flow features of the two frames of images are obtained by inputting them separately and processing them separately. and hidden features ; S5. Construct the coarse optical flow update module C, and input the optical flow features. and hidden features Simultaneously input matching features , and context features , The optical flow features of the two frames of images are obtained by inputting them separately and processing them separately. and hidden features ; S6. Construct a fine optical flow update module and input optical flow features. and hidden features Simultaneously input contextual features , The optical flow features of the two frames of images are obtained by inputting them separately and processing them separately. and hidden features ; S7. Use the convex upsampling algorithm to analyze optical flow features. Upsampling is performed to obtain the final optical flow features at the original resolution. .

2. The multi-scale optical flow estimation method based on scale experts as described in claim 1, characterized in that: Step S1 specifically includes the following steps: S11, Transfer the first frame image Second frame image Using bilinear interpolation algorithm, samples were downsampled to different resolutions. Four downsampled images of different resolutions were obtained from the two frames. and ,in , , , ; for downsampled images and Perform 1×1 convolutions separately to obtain edge features. , ; S12, Transfer the first frame image Second frame image The images are processed sequentially through a 3×3 convolution with a stride of 2, a LayerNorm layer, another 3×3 convolution with a stride of 2, and a LayerNorm layer, resulting in intermediate results A1 and A2 where the resolution of the two frames is reduced to 1 / 4 of the original image. Compare the intermediate result A1 with the edge features of the same resolution. After addition, the results are processed by two convolutional modules to obtain the high-resolution contextual features of the first frame image. ; Compare the intermediate result A2 with the edge features of the same resolution. After addition, the results are processed by two convolutional modules to obtain the high-resolution contextual features of the first frame image. ; The convolutional module contains, in sequence, a 3×3 convolutional layer with a stride of 1, a ReLU activation function, a LayerNorm layer, and another 3×3 convolutional layer with a stride of 1. S13, High-resolution contextual features , After being processed by 2×2 convolution downsampling with a stride of 2, intermediate results B1 and B2 with a resolution reduced to 1 / 8 are obtained. Compare the intermediate result B1 with the edge features of the same resolution. After addition, the results are processed sequentially through two POLA-Transformer modules. The results are divided into two paths, each of which undergoes a 1×1 convolution with a stride of 1 to obtain matching features. and context features ; Compare the intermediate result B2 with the edge features of the same resolution. After addition, the results are processed sequentially through two POLA-Transformer modules. The results are divided into two paths, each of which undergoes a 1×1 convolution with a stride of 1 to obtain matching features. and context features ; S14. Matching features and context features The features are concatenated, and the results are then subjected to 2×2 convolution downsampling with a stride of 2 to obtain an intermediate result C1 with a resolution of 1 / 16. The intermediate result C1 is then compared with edge features of the same resolution. After addition, the results are processed sequentially through two POLA-Transformer modules. The results are divided into two paths, each of which undergoes a 1×1 convolution with a stride of 1 to obtain matching features. and context features ; Matching features and context features The feature dimensions are concatenated, and the results are then processed by 2×2 convolution downsampling with a stride of 2 to obtain an intermediate result C2 with a resolution of 1 / 16. Compare the intermediate result C2 with edge features of the same resolution. After addition, the results are processed sequentially through two POLA-Transformer modules. The results are divided into two paths, each of which undergoes a 1×1 convolution with a stride of 1 to obtain matching features. and context features ; S15. Matching features and context features The features are concatenated, and the results are then subjected to 2×2 convolution downsampling with a stride of 2 to obtain an intermediate result D1 with a resolution of 1 / 16. The intermediate result D1 is then compared with edge features of the same resolution. After addition, the results are processed sequentially through two POLA-Transformer modules. The results are divided into two paths, each of which undergoes a 1×1 convolution with a stride of 1 to obtain matching features. and context features ; Matching features and context features The feature dimensions are concatenated, and the results are then processed by 2×2 convolution downsampling with a stride of 2 to obtain an intermediate result D2 with a resolution of 1 / 16. Compare the intermediate result D2 with the edge features of the same resolution. After addition, the results are processed sequentially through two POLA-Transformer modules. The results are divided into two paths, each of which undergoes a 1×1 convolution with a stride of 1 to obtain matching features. and context features .

3. The multi-scale optical flow estimation method based on scale experts as described in claim 1, characterized in that: The process of initializing optical flow and hidden features in step S2 is as follows: S21. Set the initial optical flow feature to a zero tensor, with a spatial resolution of . The channel count is 2, used to represent the initial displacement in the horizontal and vertical directions; S22. Set the initial hidden features to all-zero tensors, with a spatial resolution consistent with the initial optical flow field, i.e. The number of channels is D, which is used to store and transmit global context information and motion history information in subsequent loop iterations.

4. The multi-scale optical flow estimation method based on scale experts as described in claim 1, characterized in that: In step S3, the coarse optical flow update module A operates as follows: S31. Calculate the matching features of the first frame. Second frame matching features The dot product is calculated using the following formula: ,in The coordinates of the matching features in the first and second frames are respectively. ; The correlation of global feature similarity at a given resolution; S32. Based on the initial optical flow characteristics and global correlation characteristics Perform standard rigid body sampling to obtain local features of a 4×4 area in the optical flow pointing coordinates. Correlation; Based on the initial optical flow features and the context features of the second frame Perform standard rigid body sampling to obtain the context sampling features from the second frame to the first frame. ; S33. Correlation of local features The initial optical flow features are input into a standard motion encoder constructed using the RAFT model to extract motion features containing motion information. ; motion characteristics The features are concatenated with the initial hidden features along the feature dimension and then input into the ConvNeXt module for processing to obtain the non-occluded features. ; Then sample the context features from the second frame to the first frame. First frame context features and non-occlusion features The images are mapped to vectors using standard image embedding operations, then input into the POLA-Transformer module to capture global long-range dependencies and occlusion relationships, and output occlusion features. ; S34. Non-occluded features and The occlusion features are concatenated and fused through a 1×1 convolutional layer to generate updated hidden features. ; Hide features The new optical flow features are estimated by inputting them into the standard prediction head constructed from the RAFT model. ; S35. Repeat steps S32 to S34 once to obtain the result. The hidden features and optical flow features at a given resolution are collectively referred to as optical flow features. and hidden features .

5. The multi-scale optical flow estimation method based on scale experts as described in claim 1, characterized in that: In step S4, the coarse optical flow update module B processes the process as follows: S41, Optical flow characteristics and hidden features Its spatial resolution is magnified by a factor of 2 using bilinear interpolation, so that its resolution becomes This yields the initial optical flow feature F2' and the initial hidden feature H2' of the current level; S42. Calculate matching features and matching features The dot product is calculated using the following formula: ,in These are the coordinates of the matched features in the first and second frames, respectively. Recorded at resolution The correlation of global feature similarity; S43. Based on the initial optical flow characteristics F2' and global correlation characteristics Perform standard rigid body sampling to obtain the local feature correlation of a local 4×4 area in the optical flow pointing coordinates. Based on the initial optical flow feature F2' and the context features of the second frame. Perform standard rigid body sampling to obtain the context sampling features from the second frame to the first frame. ; S44. The local feature correlation obtained in step S43 The initial optical flow feature F2' is input into a standard motion encoder constructed using the RAFT model to extract motion features containing motion information. ; motion characteristics The feature is concatenated with the initial hidden feature H2' along the feature dimension and then fed into a standard ConvNeXt module to capture motion information of the local unoccluded region, outputting the unoccluded feature. ; Context sampling features from the second frame to the first frame Contextual features and non-occlusion features The images are mapped to vectors using standard image embedding operations and then directly input into the POLA-Transformer module to capture global long-range dependencies and occlusion relationships, outputting occlusion features. ; S45, Non-occluded features With occlusion features The features are concatenated and then fused using a 1×1 convolutional layer to generate updated hidden features. ; Hide features The new optical flow value is estimated by inputting it into the standard prediction head constructed by the RAFT model. ; S46. Repeat steps S43 to S45 once to obtain the result. The hidden features and optical flow features at a given resolution are collectively referred to as optical flow features. and hidden features .

6. The multi-scale optical flow estimation method based on scale experts as described in claim 1, characterized in that: In step S5, the coarse optical flow update module C processes the process as follows: S51, Optical flow characteristics and hidden features Its spatial resolution is magnified by a factor of 2 using bilinear interpolation algorithm, thus increasing its resolution size. This yields the initial optical flow feature F3' and the initial hidden feature H3' of the current level; S52. Calculate matching features and matching features The dot product is calculated using the following formula: ,in These are the coordinates of the matched features in the first and second frames, respectively. Recorded in Correlation of global feature similarity at different resolutions; S53. Based on the initial optical flow characteristics F3' and global correlation characteristics Perform standard rigid body sampling to obtain the local feature correlation of a local 4×4 area in the optical flow pointing coordinates. Based on the initial optical flow feature F3' and the context features of the second frame. Perform standard rigid body sampling to obtain the context sampling features from the second frame to the first frame. ; S54. The local feature correlation obtained in step S53 The initial optical flow feature F3' is input into a standard motion encoder constructed using the RAFT model to extract motion features containing motion information. ; motion characteristics The feature is concatenated with the initial hidden feature H3' along the feature dimension and then fed into a standard ConvNeXt module to capture motion information of the local unoccluded region, outputting the unoccluded feature. ; Context sampling features from the second frame to the first frame First frame context features and non-occlusion features The images are mapped to vectors using standard image embedding operations and then directly input into the POLA-Transformer module to capture global long-range dependencies and occlusion relationships, outputting occlusion features. ; S55, Non-occlusion features With occlusion features The features are concatenated and then fused using a 1×1 convolutional layer to generate updated hidden features. ; Hide features The new optical flow value is estimated by inputting it into the standard prediction head constructed by the RAFT model. ; S56. Repeat steps S53 to S55 once to obtain the result. The hidden features and optical flow features at a given resolution are collectively referred to as optical flow features. and hidden features .

7. The multi-scale optical flow estimation method based on scale experts as described in claim 1, characterized in that: In step S6, the fine optical flow update module processes the process as follows: S61, Optical flow characteristics and hidden features Its spatial resolution is magnified by a factor of 2 using bilinear interpolation algorithm, thus increasing its resolution size. This yields the initial optical flow F4' and initial hidden feature H4' for the current level; S62. At this high-resolution level, the construction of the global correlation cost volume and correlation sampling operations are not performed; instead, fine-grained updates are directly implemented. Standard rigid body sampling is performed based on the initial optical flow F4' and the context features of the second frame to obtain the context sampling features from the second frame to the first frame. ; S63. The context sampling features obtained in step S62 High-resolution contextual features of the first frame The initial optical flow F4' is concatenated with the vectors, mapped to a standard image embedding operation, and directly input into the POLA-Transformer module to capture global long-range dependencies and occlusion relationships, outputting updated hidden features, which are the hidden features. ; Hide features The new optical flow features are estimated by inputting them into the standard prediction head constructed by the RAFT model; these optical flow features are then called optical flow characteristics. .

8. The multi-scale optical flow estimation method based on scale experts as described in claim 1, characterized in that: The specific operation of step S7 is as follows: S71, Hide Features After sequential processing with 3×3 convolution, ReLU activation function, and 3×3 convolution, a 4x upsampling mask is output. This 4x upsampling mask and the optical flow features are then upsampled by 4x using a standard convex upsampling algorithm to obtain the optical flow features at the original resolution. .

Images