Optical flow estimation method and device based on depth perception and global-local collaboration

Abstract

The method comprises the following steps: 1) collecting continuous frame images; 2) building an optical flow estimation network, the input of two continuous frame images is respectively extracted by a depth perception feature encoder to obtain a fusion feature map, then zero optical flow is taken as an initial optical flow estimation, and multiple iteration optimizations are carried out: in each iteration, a self-adaptive feature alignment module is entered for feature alignment, a global-local collaborative refinement module is entered for global-local feature collaborative refinement, and an optical flow prediction module is entered for optical flow updating, the optical flow of one-half resolution of the input image is up-sampled to the original image resolution, and a final optical flow map is output; 3) training the optical flow estimation network; 4) inputting the continuous frame image pairs in a test set into the optical flow estimation network for optical flow estimation. The application alleviates the feature matching difficulty in strong occlusion and large range weak texture area, and improves the precision and robustness of optical flow estimation.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and image processing technology, and in particular to an optical flow estimation method and apparatus based on depth perception and global-local collaboration, specifically used to extract the optical flow field between consecutive frames of images in challenging scenarios such as complex geometric structures, large displacements, and occlusions. Background Technology

[0002] Optical flow estimation, aiming to calculate pixel-level motion displacement between two consecutive frames in a video sequence, is a fundamental and core task in computer vision. High-precision optical flow estimation plays a crucial role in various downstream tasks such as autonomous driving, robot navigation, video object tracking and action recognition, and 3D reconstruction. However, due to the high complexity of natural scenes, especially in areas with object self-occlusion, drastic lighting changes, large-scale motion, and weakly textured regions lacking obvious texture, traditional optical flow calculation methods, and even recent deep learning optical flow networks, still face severe challenges such as local matching ambiguity, error accumulation and propagation, and loss of details in fine structures.

[0003] Most existing technologies employ the classic convolutional neural network architecture of "feature encoding-feature matching-iterative refinement". However, when dealing with complex motion scenes, this conventional network architecture has the following obvious limitations:

[0004] First, existing feature encoders lack the ability to perceive "depth" and "geometry". Traditional feature extraction generally relies on conventional convolutional networks and standard visual Transformers. Although they can extract color and edge texture features of the image surface relatively well, they are difficult to understand the deep three-dimensional geometric structure of the scene. When encountering images with large areas of occlusion or lacking texture information, relying solely on RGB surface features can easily lead to limited feature representation capabilities.

[0005] Second, noise and errors are easily introduced during feature alignment. Current mainstream research methods generally use the warping operation to directly align the features of the next frame to the previous frame. However, in reality, in areas such as occluded edges where there are initial errors in optical flow estimation, this direct forced alignment will mix distorted noise features into subsequent modules, causing the error to be continuously amplified and propagated in multiple iterations.

[0006] Third, iterative networks struggle to simultaneously capture long-distance, large-scale displacement trends and repair localized, fine-grained movements. To address this issue, some solutions employ full-image attention mechanisms, leading to a significant increase in computational cost; while simply using small-kernel convolutions results in a limited receptive field, weakening the network's ability to handle large-scale displacements.

[0007] Patent document CN113538527A discloses a small-parameter optical flow estimation method, aiming to improve the performance of optical flow estimation by employing a six-layer pyramid network structure. This method utilizes a multi-scale pyramid descent strategy to handle object motion. Although the structure is compact and considers occlusion detection, it only uses basic convolutional modules in the feature representation stage, lacking direct perception of the scene's spatial geometric depth. When faced with targets exhibiting severe self-occlusion or large-area smooth, weak-texture translation, this patent relies solely on multi-scale RGB shallow features for matching, easily leading to holes or local matching distortions. It lacks a macroscopic hierarchical geometric prior to guide the network in understanding the scene's spatial geometric structure and depth hierarchy, making it difficult to form stable and continuous optical flow estimation in the presence of occlusion or large-area weak-texture regions.

[0008] Patent document CN107527358A discloses a dense optical flow estimation method and apparatus, which improves the stability and efficiency of estimation by combining a sparse optical flow estimation algorithm and a convolutional neural network. However, in the optical flow prediction step, this method mainly relies on traditional interpolation methods or relatively simple feature adaptive correction, and does not evaluate the safety of the matching in real time during the core feature space alignment stage. It not only struggles to filter out artifacts caused by forced distortion operations in areas of strong occlusion or motion blur, but also lacks a clear mechanism in subsequent detail correction to enable the network to first systematically grasp the global long-span displacement trend and then adjust the local boundaries in a targeted manner, resulting in the output optical flow map having insufficient sharpness at the edges of complex objects.

[0009] Therefore, a method is needed that can break through the limitations of conventional RGB surface features, introduce geometric depth-aware priors, enhance adaptive noise resistance in feature alignment, and realize the interaction between global features and local structures when processing multi-scale motion displacement, so as to effectively improve the accuracy of dense optical flow estimation in complex scenes. Summary of the Invention

[0010] The present invention aims to overcome the above-mentioned shortcomings of the prior art by proposing an optical flow estimation method and apparatus based on depth perception and global-local coordination.

[0011] The concept of this invention is to propose an end-to-end optical flow estimation architecture based on deep learning. This architecture mainly consists of a full-dimensional feature extraction stage and a successively optimized optical flow iterative prediction stage. Specifically, in the feature extraction stage, a dual-path deep-aware feature encoder is constructed for heterogeneous fusion. A large-scale pre-trained deep model is introduced to extract deep-aware features containing scene geometry, and these features are cross-fused with local high-resolution image detail features, injecting three-dimensional geometric space prior guidance into optical flow calculation. In the optical flow iterative prediction stage, the network uses the fused feature map output from the feature encoding stage to progressively optimize the optical flow through multiple iterative iterations. Each iteration comprises three main processing modules: First, an adaptive feature alignment module predicts the confidence distribution of spatial matching in real time, applying feature mapping only in reliable regions while adaptively limiting deformation feature responses in occluded edge regions, preserving the feature representation of the first frame to reduce deformation errors during iterative propagation. Next, a global-local collaborative thinning module extracts contextual features from the global field of view using a large kernel reparameterization structure, then performs secondary structure verification on local pixel boundary features using a hybrid dynamic convolution mechanism. Finally, the features processed by collaborative thinning enter the optical flow prediction module. Under gating logic, the network not only calculates the optical flow correction increment for the two output channels to accumulate in the next iteration, but also predicts the probability uncertainty parameter and generates upsampled convex combination weights for the local range of each pixel. Ultimately, a coherent high-resolution optical flow map is output.

[0012] The first aspect of this invention proposes an optical flow estimation method based on depth sensing and global-local coordination, which includes the following steps:

[0013] S1: Collect consecutive frame image pairs containing motion scenes as sample data, preprocess the sample data to obtain a sample dataset, and divide the sample dataset into training set, validation set and test set according to a set ratio;

[0014] S2: An optical flow estimation network is built based on a deep learning framework. The optical flow estimation network consists of four core modules connected in series: Deep Perception Feature Encoder (DFE), Adaptive Feature Alignment (AWM), Global-Local Co-refinement (OFRM), and Optical Flow Prediction (OFPM). The input two consecutive frames of images are first processed by the DFE module to extract fused feature maps. Then, with zero optical flow as the initial optical flow estimate, the process enters a multi-iterative optimization process. In each iteration, the input image enters the AWM module for feature alignment, the OFRM module for global-local feature co-refinement, and the OFPM module for optical flow update. After multiple iterations of optimization, the optical flow of the input image is upsampled to the original image resolution at half the resolution, and the final optical flow map is output.

[0015] S3: Input the training set into the optical flow estimation network for iterative training to obtain the trained optical flow estimation network; and use the validation set to validate the trained optical flow estimation network to obtain the final optical flow estimation network.

[0016] S4: Input the continuous frame images in the test set into the final optical flow estimation network. The DFE module extracts the fused feature map, which is then refined through multiple iterations by the AWM, OFRM, and OFPM modules. Finally, it is upsampled to the original resolution to complete the optical flow estimation of the moving scene in the test set.

[0017] More specifically, step S1 includes:

[0018] S1.1: Acquire consecutive frame image pairs containing different motion types and scene complexities, and obtain the ground truth optical flow map corresponding to each image pair as a supervision label;

[0019] S1.2: Perform data augmentation processing on the sample images, including random cropping, random flipping, and color jittering operations;

[0020] S1.3: Divide the sample dataset into training, validation and test sets according to the set ratio, adjust the images to a fixed size and perform normalization processing.

[0021] More specifically, step S2 includes:

[0022] S2.1: Construct a depth-aware feature encoder (DFE), which includes a depth-aware feature encoding path, an image feature encoding path, and a feature fusion unit, which respectively perform dual-path feature extraction and fusion;

[0023] S2.2: Construct an adaptive feature alignment module (AWM), which includes a spatial deformation unit, a confidence estimation unit, and an adaptive weighted gating unit to perform confidence-based feature alignment;

[0024] S2.3: Construct a global-local collaborative refinement module (OFRM), which includes a global context unit, a local fine-grained perception unit, a dynamic convolution and enhancement unit, and a feature output layer, to perform feature refinement by first outlining and then examining details;

[0025] S2.4: Construct an optical flow prediction module (OFPM), which includes a feature decoding layer, a prediction head unit, and a convex combination upsampling unit to predict optical flow update amount and uncertainty.

[0026] More specifically, step S2.1 includes:

[0027] S2.1.1: Constructing a deep-aware feature encoding path; consisting of a DINOv3 encoder based on the Depth-Anything-3 deep-aware model and a DPT feature processing head; the DINOv3 encoder is used to extract the feature representations of the fifth, seventh, ninth, and eleventh layers from the input image as features of each intermediate layer, which include image patch labels and global labels; the DPT feature processing head is connected after the DINOv3 encoder, firstly separating and projecting the extracted features, adjusting them to a uniform channel dimension and spatial resolution, and then upsampling and fusing them step by step from the fourth layer features to the first layer features through residual convolutional units, finally outputting deep-aware features containing multi-scale information;

[0028] S2.1.2: Constructing the image feature encoding path; this path is composed of ResNet18Deconv, including an encoding part and a decoding part; the encoding part is composed of four levels of residual convolutional units connected in series, used to downsample the input image and extract four levels of features, with the channel dimension doubling sequentially; the decoding part starts from the fourth level of features, upsamples through deconvolutional layers, and adds them element-wise with the features of the corresponding level of the encoding part through skip connections. The added features are refined by residual convolutional units until they are restored to the first level of resolution, generating image structural detail features;

[0029] S2.1.3: Construct a feature fusion unit; this unit consists of a feature stitching layer and a convolutional fusion layer; the feature stitching layer stitches together depth-sensing features and image structural detail features in the channel dimension, and the convolutional fusion layer performs convolutional projection on the stitched features to map them into a fusion feature map with a uniform number of channels.

[0030] More specifically, step S2.2 includes:

[0031] S2.2.1: Construct a spatial deformation unit; the input of this unit is connected to the current optical flow estimation map and the second frame fused feature map. For the first iteration, the current optical flow estimation map is the input zero optical flow; this unit uses the current optical flow estimation map to calculate the sampling coordinates of each pixel in the second frame fused feature map, and performs spatial deformation through bilinear interpolation, finally outputting an aligned feature map;

[0032] S2.2.2: Construct the confidence estimation unit; this unit consists of an input stitching layer, three convolutional layers, a batch normalization layer, and a SiLU activation layer connected in series; the input stitching layer stitches together four items along the channel dimension: the first frame fused feature map, the aligned feature map, the pixel-wise absolute difference map between the first frame fused feature map and the aligned feature map, and the current optical flow estimation map; the terminal convolutional layer is connected to the Sigmoid activation function to output a pixel-wise confidence map with values ​​ranging from zero to one;

[0033] S2.2.3: Construct an adaptive weighted gating unit; the input of this unit receives a confidence map, an alignment feature map, and a first frame fused feature map; this unit uses the confidence map as a gating coefficient to perform a pixel-by-pixel weighted summation operation on the alignment feature map and the first frame fused feature map, that is, for each pixel position, the alignment feature map is weighted by the confidence value, and the first frame fused feature map is weighted by one minus the confidence value, and the two are added together to output an adaptive alignment feature map.

[0034] More specifically, step S2.3 includes:

[0035] S2.3.1: Constructing a global context unit; consisting of stacked large-kernel reparameterized convolutional blocks (RepConvBlock), adaptive global average pooling layers, compressed convolutional layers, and upsampling layers connected in series; wherein, RepConvBlock internally contains residual depthwise separable convolutions, dilated reparameterized blocks, batch normalization layers, and channel attention units; this unit is used to extract adaptive aligned feature maps and generate global context features consistent with the input resolution; a multi-branch parallel structure is adopted during the training phase to enhance feature extraction capability, and during the inference phase, the parameters of multiple branches are equivalently fused into a single convolutional kernel to achieve structural reparameterization and accelerate inference;

[0036] S2.3.2: Construct a local fine-grained perception unit; composed of context-mixed dynamic convolutional blocks; the input of this unit has a splicing and fusion layer, which is used to splice and fuse the adaptive aligned feature map with the global context features; the unit has a query vector generator and a key vector generator, which generate query vectors and key vectors from the spliced ​​and fused features and the global context features, respectively, and generate global association weights through dot product operation;

[0037] S2.3.3: Constructing a dynamic convolution and enhancement unit; including a dynamic parameter projection layer, a grouped convolutional layer, a positional encoding stacking layer, and an SE gated unit; the dynamic parameter projection layer maps the global correlation weights to two sets of dynamic convolutional kernel parameters corresponding to small kernels and large kernels respectively; the grouped convolutional layer uses the dynamic convolutional kernel parameters to perform grouped convolution on the adaptively aligned feature maps; the positional encoding stacking layer contains an internal dilated reparameterization block, and adds the grouped convolution output to the local enhancement positional encoding generated by the dilated reparameterization block, and finally modulates the output of the local refinement feature through the SE gated unit;

[0038] S2.3.4: Construct the feature output layer; The feature output layer is connected to the end of the local fine-grained perceptual unit. The local fine-grained features output by the stacked context-mixed dynamic convolutional blocks are batch normalized and convolutionally projected to obtain the final co-refined features.

[0039] More specifically, step S2.4 includes:

[0040] S2.4.1: Construct the feature decoding layer; This layer is connected after the local fine perception unit and is used to maintain and update the hidden state in the iterative loop process. In the first iteration, the state is initialized to a zero tensor, and its output is connected to the optical flow prediction head and the upsampling weight prediction head, respectively.

[0041] S2.4.2: Constructing the prediction head unit; The optical flow prediction head consists of two convolutional layers with different kernel sizes connected in series, used to output the optical flow update amount of the two channels and additional optical flow uncertainty parameters; The upsampling weight prediction head consists of convolutional layers, used to output upsampling mask weights for a set number of channels;

[0042] S2.4.3: Construct a convex combination upsampling unit; the input end receives the optical flow update, the current optical flow estimation map and the upsampling mask weights, accumulates the optical flow update to the current optical flow estimation map, and uses the mask weights to perform a weighted convex combination calculation of the local spatial neighborhood of the accumulated low-resolution optical flow to achieve double upsampling.

[0043] More specifically, step S3 includes:

[0044] S3.1: Set the network training parameters, including batch size, initial learning rate, maximum number of iterations, and number of iterations; use the AdamW optimizer to iteratively train the optical flow estimation network;

[0045] S3.2: Input the images in the training set into the optical flow estimation network in batches, and iterate through the DFE module, AWM module, OFRM module and OFPM module in sequence multiple times to obtain the optical flow prediction results and uncertainty parameters output in each iteration;

[0046] S3.3: Construct a hybrid loss function; For the optical flow prediction results output in each iteration, calculate the endpoint error between the optical flow prediction results and the true optical flow map, and construct a normalized flow loss by combining the uncertainty parameters. By weighting and summing the losses of each iteration according to the exponential decay weight, the prediction results closer to the final iteration are given a larger weight, and the final hybrid loss function value is obtained.

[0047] S3.4: Based on the mixed loss function value, the gradient is calculated and all learnable network parameters are updated through the backpropagation algorithm; training is automatically stopped when the maximum number of iterations is reached, thus obtaining a trained optical flow estimation network model;

[0048] S3.5: Use the validation set to perform a comprehensive performance evaluation of the intermediate archived models during the training process, and select the model parameters with the smallest comprehensive endpoint error on the validation set as the final optical flow estimation network to complete the training.

[0049] Secondly, the present invention proposes an optical flow estimation device based on depth perception and global-local collaboration, including a memory and a processor. The memory stores executable code, and when the processor executes the executable code, it is used to implement the optical flow estimation method based on depth perception and global-local collaboration of the present invention.

[0050] Thirdly, the present invention also provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the optical flow estimation method based on depth perception and global-local coordination of the present invention.

[0051] The innovation of this invention is:

[0052] (1) Depth-Perception-Driven Heterogeneous Feature Encoding Mechanism. Addressing the shortcomings of existing feature extractors, such as a lack of 3D scene structure perception and a tendency to produce severe matching errors in occluded and weakly textured regions, this invention integrates a large-scale pre-trained depth-perception model feature extractor into the optical flow inference process. Instead of relying solely on the RGB visual features of the image surface, it extracts geometric depth priors that imply geometric constraints and depth parallax, and then cross-fuses these priors with the acquired image structural detail features using a multi-space cascading mechanism. This geometric and texture dual-chain heterogeneous convergence layer effectively improves the optical flow network's ability to explore complex spatial geometric perspectives and their multi-layered occlusion relationships.

[0053] (2) Confidence-guided soft adaptive feature alignment. To address the problem of significant noise and error accumulation caused by direct reliance on historical optical flow guidance and strong spatial deformation operations in traditional iterative optical flow networks, this scheme designs a feature matching reliability evaluation mechanism for the matching results. It calculates reliable confidence-based gating weights by combining the feature errors of the two analysis frames before and after spatial projection with the forward displacement. Based on the gating logic of this confidence map, soft aggregation is performed: the aligned feature map of the higher confidence region is weighted and combined with the fused feature map of the first frame in the lower confidence region, such as areas with obvious occlusion. This adaptive gating mechanism effectively reduces common distortions and artifacts at occlusion edges and prevents the extraction error from diverging backward during iteration.

[0054] (3) Global-Local Collaborative Feature Iterative Refinement. To address the limitations of single small-kernel convolution computation and the high computational cost and limited performance of global self-attention mechanisms when handling displacements at different scales, this method designs a collaborative network structure that combines global context guidance with local fine-grained perception. Specifically, a large-kernel reparameterization structure is used to efficiently acquire global spatial context features with a wide field of view. Subsequently, feature fusion operations are used to dynamically generate corresponding association weights for each location, and then dynamic kernel parameters are assigned to different image regions to refine detailed features.

[0055] The beneficial effects of this invention are as follows:

[0056] (1) Improved optical flow estimation's ability to resist complex visual interferences such as occlusion, motion blur, and smooth weak texture. By incorporating depth perception priors, the algorithm of this invention generates a robust spatial constraint on the depth continuity and structural boundaries of the scene, thereby reducing the disorder and chaos in the feature matching stage.

[0057] (2) Improved the robustness of the optical flow estimation process in multiple iterations. This scheme effectively reduces error distortion and maintains the stability of the long-term iterative update state by introducing an adaptive feature gating alignment mechanism. It not only suppresses the drift of the outer point, but also makes the overall profile of the output optical flow smoother and clearer.

[0058] (3) A balance between prediction accuracy and inference efficiency has been achieved. Thanks to the equivalent fusion deployment of collaborative structure design and large kernel feature reparameterization inference stage, good prediction accuracy has been achieved while the computational burden has been well controlled. Under high resolution conditions, the controllability of computational overhead can still be maintained. Attached Figure Description

[0059] Figure 1 This is a schematic diagram of the overall process of the method of the present invention.

[0060] Figure 2 This is a diagram of the overall architecture of the optical flow estimation network of the present invention.

[0061] Figure 3 This is a schematic diagram of the structure of the depth-sensing feature encoder (DFE) of the present invention.

[0062] Figure 4 This is a schematic diagram of the adaptive feature alignment module (AWM) of the present invention.

[0063] Figure 5 This is a schematic diagram of the Global-Local Collaborative Refinement Module (OFRM) of the present invention.

[0064] Figure 6 This is a schematic diagram of the optical flow prediction module (OFPM) of the present invention. Detailed Implementation

[0065] To make the technical solution of the present invention clearer and more complete, the embodiments of the present invention will be described in detail below with reference to the accompanying drawings. These embodiments are for illustrative purposes only and are not intended to limit the present invention. Based on the embodiments of the present invention, any other equivalent modifications or substitutions made by those skilled in the art without creative effort fall within the protection scope of the present invention.

[0066] Example 1

[0067] This embodiment relates to an optical flow estimation method based on depth perception and global-local collaboration, as detailed in the appendix. Figures 1 to 6 To elaborate on this, the steps constitute the following sections:

[0068] S1: Collect consecutive frame image pairs containing motion scenes as sample data, preprocess the sample data to obtain a sample dataset, and divide the sample dataset into training set, validation set and test set according to a set ratio.

[0069] More specifically, step S1 includes:

[0070] S1.1: Acquire consecutive frame image pairs containing different motion types and scene complexities, and obtain the ground truth optical flow map corresponding to each image pair as a supervision label. Establish a basic sequence set and associate it with a known, rigorously defined real optical flow field coordinate map as a supervision signal.

[0071] S1.2: Perform data augmentation on the sample images, including random cropping, random flipping, and color jittering. Specifically, cropping, diagonal and mirror flipping in the same direction are performed before inputting the tensor to the network, and perturbations are added to the color space to provide an anti-overfitting and enhanced training environment.

[0072] S1.3: Divide the sample dataset into training, validation, and test sets according to a set ratio. Adjust the images to a fixed size and perform normalization. Set a fixed resolution normalization operation to map pixels to a zero-centered safety threshold.

[0073] S2: An optical flow estimation network is built based on a deep learning framework. The optical flow estimation network consists of four core modules connected in series: a Deep Aware Feature Encoder (DFE), an Adaptive Feature Alignment Module (AWM), a Global-Local Co-refinement Module (OFRM), and an Optical Flow Prediction Module (OFPM). Two consecutive input images are first processed by the DFE module to extract fused feature maps. Then, using zero optical flow as the initial optical flow estimate, a multi-iterative optimization process is initiated: in each iteration, the input image sequentially enters the AWM module for feature alignment, the OFRM module for global-local feature co-refinement, and the OFPM module for optical flow update. After multiple iterations, the optical flow is upsampled from half the resolution of the input image to the original image resolution, outputting the final optical flow map.

[0074] More specifically, step S2 includes:

[0075] S2.1: Construct a depth-aware feature encoder (DFE), which includes a depth-aware feature encoding path, an image feature encoding path, and a feature fusion unit, which respectively perform dual-path feature extraction and fusion.

[0076] More specifically, step S2.1 includes:

[0077] S2.1.1: Constructing the Deep-Aware Feature Encoding Path. This consists of a DINOv3 encoder based on the Depth-Anything-3 deep-aware model and a DPT feature processing head. The DINOv3 encoder extracts feature representations from layers 5, 7, 9, and 11 of the input image as intermediate layer features. These include image patch labels and global labels. The DPT feature processing head, connected after the DINOv3 encoder, first separates and projects the extracted features, adjusting them to a uniform channel dimension and spatial resolution. Then, it uses residual convolutional units to upsample and fuse features from layer 4 to layer 1, ultimately outputting deep-aware features containing multi-scale information. The specific formula represents a multi-level fusion process:

[0078]

[0079] in, Indicates to the first Deep-sensory features generated recursively layer by layer express Convolution projection operation, Represents a residual convolution unit. This represents a feature concatenation operation along the channel dimension. This indicates a feature upsampling operation. This indicates the first number extracted from the DINOv3 encoder. Layer features. The final product is a feature representation with geometric information.

[0080] S2.1.2: Constructing the image feature encoding path. This path is constructed using ResNet18Deconv and includes an encoding part and a decoding part. The encoding part consists of four levels of residual convolutional units connected in series, used to downsample the input image and extract four levels of features, with the channel dimension doubling sequentially. The decoding part starts from the fourth level of features, upsamples them through deconvolutional layers, and adds them element-wise with the features of the corresponding layers in the encoding part through skip connections. The added features are refined by residual convolutional units until they are restored to the first level of resolution, generating detailed image structural features:

[0081]

[0082] in, Indicates the first High-resolution image structural detail features This represents the residual convolution operation. express Deconvolution upsampling operation. This generates structural features with rich surface texture.

[0083] S2.1.3: Constructing the Feature Fusion Unit. This unit consists of a feature concatenation layer and a convolutional fusion layer. The feature concatenation layer concatenates depth-aware features with image structural detail features along the channel dimension. The convolutional fusion layer performs convolutional projection on the concatenated features, mapping them into a fused feature map with a uniform number of channels.

[0084]

[0085] in, This represents the fused feature map of the first frame after mapping. and These represent the hierarchical features of the first frame output by the depth-sensing extraction path and the image structure extraction path, respectively. Similarly, the fused feature map of the second frame is generated, laying a heterogeneous foundation for the subsequent feature matching stage.

[0086] S2.2: Construct an Adaptive Feature Alignment Module (AWM), which includes a spatial deformation unit, a confidence estimation unit, and an adaptive weighted gating unit to perform confidence-based feature alignment.

[0087] More specifically, step S2.2 includes:

[0088] S2.2.1: Constructing the Spatial Deformation Unit. This unit's input is connected to the current optical flow estimation map and the second frame fused feature map. For the first iteration, the current optical flow estimation map is the input zero optical flow. This unit uses the current optical flow estimation map to calculate the sampling coordinates of each pixel in the second frame fused feature map, and performs spatial deformation through bilinear interpolation, outputting an aligned feature map.

[0089]

[0090]

[0091] in, Indicates the spatial mapping coordinates used for sampling in the second frame. Represents standard two-dimensional grid coordinates. This represents the optical flow estimation map output from the previous iteration (or the initial value of zero optical flow for the first iteration). This represents the aligned feature map after extraction and deformation. Represents the fused feature map of the second frame, symbol This represents the bilinear sampling operator.

[0092] S2.2.2: Constructing the confidence estimation unit. This unit consists of an input concatenation layer, three convolutional layers, a batch normalization layer, and a SiLU activation layer connected in series. The input concatenation layer concatenates four items along the channel dimension: the first frame fused feature map, the aligned feature map, the pixel-wise absolute difference map between the first frame fused feature map and the aligned feature map, and the current optical flow estimation map. The terminal convolutional layer is connected to a Sigmoid activation function to output a pixel-wise confidence map with values ​​ranging from zero to one.

[0093]

[0094] in, This represents a pixel-by-pixel confidence map with values ​​between zero and one. This represents the Sigmoid activation function. This represents a regular convolution operation. This indicates the calculation of the absolute difference per pixel.

[0095] S2.2.3: Construct an adaptive weighted gating unit. This unit receives a confidence map, an aligned feature map, and a first-frame fused feature map as inputs. Using the confidence map as a gating coefficient, this unit performs a pixel-by-pixel weighted summation operation on the aligned feature map and the first-frame fused feature map. That is, for each pixel position, the aligned feature map is weighted by the confidence value, and the first-frame fused feature map is weighted by subtracting the confidence value from the first-frame fused feature map. The sum of the two outputs an adaptive aligned feature map.

[0096]

[0097] in, Represents the adaptive alignment feature map, symbol This represents element-wise multiplication of matrices. This design effectively blocks artifacts caused by deformation operations in occluded areas.

[0098] S2.3: Construct a global-local collaborative refinement module (OFRM), which includes a global context unit, a local fine-grained perception unit, a dynamic convolution and enhancement unit, and a feature output layer, to perform feature refinement by first outlining and then examining details.

[0099] More specifically, step S2.3 includes:

[0100] S2.3.1: Constructing the Global Context Unit. This unit consists of stacked large-kernel reparameterized convolutional blocks (RepConvBlock), adaptive global average pooling layers, compressed convolutional layers, and upsampling layers, sequentially connected in series. The RepConvBlock internally contains residual depthwise separable convolutions, dilated reparameterized blocks, batch normalization layers, and channel attention units. This unit is used to extract adaptively aligned feature maps and generate global context features consistent with the input resolution.

[0101]

[0102] in, This represents the global context features extracted by the global context unit. This indicates a bilinear interpolation upsampling operation. This indicates an adaptive global average pooling operation. RepConvBlock represents a stacked, large-kernel reparameterized convolutional block. During the inference phase, it effectively merges the parameters of multiple branches into a single convolutional kernel, achieving structural reparameterization to accelerate inference.

[0103] S2.3.2: Constructing a Local Fine-grained Perception Unit. This unit consists of context-blended dynamic convolutional blocks. The input of this unit includes a concatenation and fusion layer to concatenate and fuse the adaptively aligned feature map with the global context features. Internally, the unit includes a query vector generator and a key vector generator, which generate query vectors and key vectors from the concatenated and fused feature vectors and global context features, respectively. Global association weights are then generated through pooling and dot product operations.

[0104]

[0105]

[0106] in, and These represent the generated query vector and key vector, respectively. and This represents the feature mapping operation required to generate the corresponding vector. This represents the feature vector obtained by concatenating and fusing the adaptively aligned feature map with the global context features. This represents the calculated global association weight. This represents the Softmax normalization function. The channel dimension scaling factor represents the feature vector.

[0107] S2.3.3: Constructing the Dynamic Convolution and Enhancement Unit. This unit includes a dynamic parameter projection layer, a grouped convolutional layer, a positional encoding stacking layer, and an SE-gated unit. The dynamic parameter projection layer maps the global correlation weights to two sets of dynamic convolutional kernel parameters, corresponding to a small kernel and a large kernel, respectively. The grouped convolutional layer uses the mapped dynamic convolutional kernel parameters to perform grouped convolutions on the adaptively aligned feature map. The positional encoding stacking layer contains an internal dilated reparameterization block and adds the grouped convolution output to the local enhancement positional encoding generated by this dilated reparameterization block based on the adaptively aligned feature map. The added features are modulated by the SE-gated unit to output locally refined features:

[0108]

[0109]

[0110] in, This represents the intermediate local features after processing by convolution with dynamic parameters and positional encoding. This indicates a grouped convolution operation performed using dynamically allocated kernel parameters. This represents the generated local enhancement location code. This represents the localized refinement features after modulation by the SE-gated unit. Indicates SE gated unit, This represents the SiLU nonlinear activation function.

[0111] S2.3.4: Construct the feature output layer. This layer is connected to the end of the local fine-grained perceptual unit and is used to batch normalize and convolutionally project the local fine-grained features output by the stacked context-mixed dynamic convolutional blocks to obtain the final co-refined features.

[0112] S2.4: Construct an optical flow prediction module (OFPM), which includes a feature decoding layer, a prediction head unit, and a convex combination upsampling unit to predict optical flow update amount and uncertainty.

[0113] More specifically, step S2.4 includes:

[0114] S2.4.1: Constructing the Feature Decoding Layer. This layer, connected after the Local Fine-grained Perceptual Unit (RFU), mainly consists of convolutional layers forming a state update logic similar to that of a gated recurrent unit. This layer maintains the hidden state during the iterative loop and updates the current hidden state using the collaboratively refined features of the current iteration input and the hidden state of the previous iteration. For the first iteration, the hidden state is initialized to a shared zero tensor.

[0115]

[0116] in, This represents the implicit state output obtained after the current iteration step. This represents the historical implicit state passed down from the previous iteration. This represents the feature representation that currently flows in through the global-local collaborative refinement module. This represents a transformation convolution operation that performs state fusion computation of the gated layer.

[0117] S2.4.2: Constructing the prediction head unit. The optical flow prediction head consists of concatenated convolutional layers with different kernel sizes, used to output the optical flow update amounts for two channels and additional optical flow uncertainty parameters:

[0118]

[0119] in, This represents the residual of the predicted two-channel optical flow update. and These represent the predicted optical flow uncertainty parameters in the horizontal and vertical dimensions, respectively. This represents the optical flow prediction head unit.

[0120] The upsampling weight prediction head consists of convolutional layers and is used to predict upsampling mask weights for a set number of output channels from the hidden state.

[0121] S2.4.3: Construct a convex combination upsampling unit. This unit receives the optical flow update, the current optical flow estimation map, and the upsampling mask weights at its input. This unit accumulates the optical flow update into the current optical flow estimation map.

[0122]

[0123] in, This represents the current optical flow estimation map generated by superimposing the optical flow update amount and the historical optical flow in the current iteration step.

[0124] The predicted mask weights are used to perform a weighted Softmax convex combination calculation of the local spatial neighborhood of the accumulated low-resolution optical flow, achieving a 2x upsampling, and finally reconstructing and outputting a high spatial resolution optical flow map.

[0125] S3: Input the training set into the optical flow estimation network for iterative training to obtain the trained optical flow estimation network; and use the validation set to validate the trained optical flow estimation network to obtain the final optical flow estimation network.

[0126] More specifically, step S3 includes:

[0127] S3.1: Set the network training parameters, including batch size, initial learning rate, maximum number of iterations, and number of iterations. Use the AdamW optimizer to iteratively train the optical flow estimation network.

[0128] S3.2: Input the images in the training set into the optical flow estimation network in batches, and iterate through the DFE module, AWM module, OFRM module and OFPM module in sequence to obtain the optical flow prediction results and uncertainty parameters output in each iteration.

[0129] S3.3: Constructing the hybrid loss function. For the optical flow prediction result output in each iteration, calculate the endpoint error between the prediction result and the true optical flow map, and construct the normalized flow loss by combining the uncertainty parameter; by weighting and summing the losses of each iteration according to exponential decay weights, the prediction result closer to the final iteration receives a larger weight, as expressed by the formula:

[0130]

[0131] in, This represents the final value of the hybrid loss function used in training the optical flow estimation network. This represents the upper limit of the number of iterations required to solve the problem. Indicates the current iteration number. It is a weight decay hyperparameter used for the exponential sequence decrease penalty (with a limit value of less than one). This represents the calculated optical flow endpoint error distance loss term. This indicates the true optical flow field label that provides physical supervision constraints. This represents the normalized flow loss term obtained based on the negative log-likelihood distribution. Unified reference in The uncertainty prediction adjustment parameter generated in the next iteration.

[0132] S3.4: Based on the hybrid loss function value, the gradient is calculated and all learnable network parameters are updated using the backpropagation algorithm. Training automatically stops when the maximum number of iterations is reached, thus obtaining the trained optical flow estimation network model.

[0133] S3.5: Use the validation set to perform a comprehensive performance evaluation of the intermediate archived models during the training process, and select the model parameters with the smallest comprehensive endpoint error on the validation set as the final optical flow estimation network to complete the training process.

[0134] S4: Input the continuous frame images in the test set into the final optical flow estimation network. The DFE module extracts the fused feature map, which is then refined through multiple iterations by the AWM, OFRM, and OFPM modules. Finally, it is upsampled to the original resolution to complete the optical flow estimation of the moving scene in the test set.

[0135] To verify the superiority of the optical flow estimation method proposed in this invention, extensive quantitative experiments and validations were conducted on a series of widely used standard datasets. The specific experimental procedure is as follows: During the network training phase, the model was first pre-trained on large-scale synthetic general optical flow datasets such as FlyingChairs (Chairs) and FlyingThings3D (Things), enabling the network to initially establish feature matching and prediction capabilities for multi-scale displacement and boundary deformation. Subsequently, based on the validation requirements of different real-world scenarios, supervised fine-tuning training was further performed on the corresponding target datasets. In the testing and evaluation phase, this invention was mainly applied to test datasets such as Sintel (containing complete Clean and Final test sequences) and KITTI autonomous driving scenarios for validation. The experiment used the average End-Point Error (EPE, the smaller the value, the more accurate the prediction and the better the performance) as the main performance evaluation metric for the Sintel dataset; and used the percentage of outliers (Fl-all, the smaller the value, the better the performance) as the evaluation metric for the KITTI dataset. The experiment also compared and recorded the number of model parameters and the single-frame inference time of each method during the inference process.

[0136] Table 1 compares the optical flow estimation method of this invention with other methods on the Sintel and KITTI datasets, and Table 2 compares the parameters and running time of the optical flow estimation method of this application with other methods.

[0137] Table 1

[0138]

[0139] Table 2

[0140]

[0141] As shown in Tables 1 and 2, the present invention achieves competitive performance on mainstream benchmark datasets such as Sintel and KITTI. A separate comparison of the lightweight methods in Table 1 shows that the method of this application achieves low mean endpoint errors of 1.48 and 2.54 on the Sintel Clean and Final datasets, respectively, and a full-pixel F1 error rate percentage of 14.3% on the KITTI dataset.

[0142] Furthermore, Table 2 shows the overall performance of the method of this invention compared to many mainstream large-scale optical flow schemes in terms of parameter quantity and runtime. The runtime was measured at a test resolution of 540×960 on an NVIDIA 3090 GPU. The algorithm of this invention achieves high accuracy on Sintel and KITTI, while its model parameter quantity is 34.8M, and its single-frame inference time is only 131.3 milliseconds, approaching or surpassing many classic large models. This significantly outperforms the very few previously recognized large-scale pre-training schemes with excellent test accuracy and outstanding generalization ability, such as the CroCoFlow model with 437.4M absolute parameters and an extremely long inference time of 6422.0 milliseconds. Although some large networks do exhibit outstanding accuracy, their excessively large computational scale and extremely high memory usage completely exceed the requirements of real-time tasks. In contrast, this invention maintains high accuracy and consistent optical flow prediction without requiring excessive hardware computing power or large single-frame inference time constraints.

[0143] Example 2

[0144] This embodiment provides an optical flow estimation device based on depth perception and global-local collaboration, including a memory and a processor. The memory stores executable code, and when the processor executes the executable code, it is used to implement the optical flow estimation method based on depth perception and global-local collaboration of Embodiment 1.

[0145] Example 3

[0146] This embodiment provides a computer-readable storage medium on which a program is stored. When the program is executed by a processor, it implements the optical flow estimation method based on depth perception and global-local coordination of Embodiment 1.

[0147] The embodiments described in this specification are merely examples of implementations of the inventive concept. The scope of protection of this invention should not be considered as limited to the specific forms stated in the embodiments. The scope of protection of this invention also extends to equivalent technical means that can be conceived by those skilled in the art based on the inventive concept.

Claims

1. An optical flow estimation method based on depth perception and global-local coordination includes the following steps: S1: Collect consecutive frame image pairs containing motion scenes as sample data, preprocess the sample data to obtain a sample dataset, and divide the sample dataset into training set, validation set and test set according to a set ratio; S2: An optical flow estimation network is built based on a deep learning framework. The optical flow estimation network consists of a Deep Aware Feature Encoder (DFE) module, an Adaptive Feature Alignment (AWM) module, a Global-Local Co-refinement (OFRM) module, and an Optical Flow Prediction (OFPM) module connected in series. The input two consecutive frames of images are first processed by the DFE module to extract fused feature maps. Then, with zero optical flow as the initial optical flow estimate, the process enters a multi-iterative optimization process. In each iteration, the input image enters the AWM module for feature alignment, the OFRM module for global-local feature co-refinement, and the OFPM module for optical flow update. After multiple iterations of optimization, the optical flow of the input image is upsampled to the original image resolution at half the resolution, and the final optical flow map is output. The DFE module contains parallel deep-aware feature encoding paths based on Depth-Anything-3 and ResNet18Deconv, as well as a feature fusion unit for fusing the outputs of the two paths. Its output is connected to the AWM module. The AWM module contains a spatial deformation unit, a confidence estimation unit, and an adaptive weighted gating unit. Its output is connected to the OFRM module. The OFRM module contains cascaded global context units and local fine-grained perception units, as well as dynamic convolution and enhancement units. Its output is connected to the OFPM module. The OFPM module contains a feature decoding layer, an optical flow and weight prediction head unit, and a convex combination upsampling unit. Its output is connected back to the AWM module for the next iteration. S3: Input the training set into the optical flow estimation network for iterative training to obtain the trained optical flow estimation network; The trained optical flow estimation network is then validated using a validation set to obtain the final optical flow estimation network. S4: Input the continuous frame images in the test set into the final optical flow estimation network. The DFE module extracts the fused feature map, which is then refined through multiple iterations by the AWM, OFRM, and OFPM modules. Finally, it is upsampled to the original resolution to complete the optical flow estimation of the moving scene in the test set.

2. The optical flow estimation method based on depth perception and global-local coordination as described in claim 1, characterized in that, Step S1 includes: S1.1: Acquire consecutive frame image pairs containing different motion types and scene complexities, and obtain the ground truth optical flow map corresponding to each image pair as a supervision label; S1.2: Perform data augmentation processing on the sample images, including random cropping, random flipping, and color jittering operations; S1.3: Divide the sample dataset into training, validation and test sets according to the set ratio, adjust the images to a fixed size and perform normalization processing.

3. The optical flow estimation method based on depth perception and global-local coordination as described in claim 1, characterized in that, Step S2 includes: S2.1: Construct a depth-aware feature encoder (DFE). The DFE module includes a depth-aware feature encoding path, an image feature encoding path, and a feature fusion unit, which respectively perform dual-path feature extraction and fusion. S2.2: Construct an adaptive feature alignment module (AWM). The AWM module includes a spatial deformation unit, a confidence estimation unit, and an adaptive weighted gating unit, and performs confidence-based feature alignment. S2.3: Construct a global-local collaborative refinement module OFRM. The OFRM module includes a global context unit, a local fine perception unit, a dynamic convolution and enhancement unit, and a feature output layer, which performs feature refinement by first overviewing and then examining details. S2.4: Construct the optical flow prediction module OFPM. The OFPM module includes a feature decoding layer, a prediction head unit, and a convex combination upsampling unit to predict the optical flow update amount and uncertainty.

4. The optical flow estimation method based on depth perception and global-local coordination as described in claim 3, characterized in that, Step S2.1 specifically includes: S2.1.1: Constructing a deep-aware feature encoding path; consisting of a DINOv3 encoder and a DPT feature processing head for the deep-aware model; the DINOv3 encoder is used to extract the feature representations of the fifth, seventh, ninth, and eleventh layers from the input image as features of each intermediate layer, which include image patch labels and global labels; the DPT feature processing head is connected after the DINOv3 encoder, first separating and projecting the extracted features to adjust them to a uniform channel dimension and spatial resolution, and then upsampling and fusing them step by step from the fourth layer features to the first layer features through residual convolution units, finally outputting deep-aware features containing multi-scale information; S2.1.2: Constructing the image feature encoding path; composed of ResNet18Deconv modules, including encoding and decoding parts; the encoding part consists of four levels of residual convolutional units connected in series, used to downsample the input image and extract four levels of features, with the channel dimension doubling sequentially; the decoding part starts from the fourth level of features, upsamples through deconvolutional layers, and adds them element-wise with the features of the corresponding level of the encoding part through skip connections. The added features are refined by residual convolutional units until they are restored to the first level of resolution, generating image structural detail features; S2.1.3: Construct a feature fusion unit; consisting of a feature stitching layer and a convolutional fusion layer; the feature stitching layer stitches together depth-sensing features and image structural detail features in the channel dimension, and the convolutional fusion layer performs convolutional projection on the stitched features to map them into a fusion feature map with a uniform number of channels.

5. The optical flow estimation method based on depth perception and global-local coordination as described in claim 3, characterized in that, Step S2.2 specifically includes: S2.2.1: Construct a spatial deformation unit, whose input is connected to the current optical flow estimation map and the fused feature map of the second frame; calculate the sampling coordinates of each pixel in the fused feature map of the second frame using the current optical flow estimation map, and perform spatial deformation through bilinear interpolation to output an aligned feature map; S2.2.2: Construct a confidence estimation unit, which consists of an input stitching layer, three convolutional layers, a batch normalization layer, and a SiLU activation layer connected in series. The input stitching layer stitches together four items along the channel dimension: the first frame fused feature map, the aligned feature map, the pixel-wise absolute difference map between the first frame fused feature map and the aligned feature map, and the current optical flow estimation map. The terminal convolutional layer is connected to a Sigmoid activation function to output a pixel-wise confidence map with values ​​ranging from zero to one. S2.2.3: Construct an adaptive weighted gating unit, whose input receives a confidence map, an alignment feature map, and a first frame fused feature map; using the confidence map as a gating coefficient, perform a pixel-wise weighted summation operation on the alignment feature map and the first frame fused feature map, that is, for each pixel position, the alignment feature map is weighted by the confidence value, and the first frame fused feature map is weighted by one minus the confidence value, and the two are added together to output the adaptive alignment feature map.

6. The optical flow estimation method based on depth perception and global-local coordination as described in claim 3, characterized in that, Step S2.3 specifically includes: S2.3.1: Constructing a global context unit; consisting of stacked large-kernel reparameterized convolutional blocks RepConvBlock, adaptive global average pooling layers, compressed convolutional layers, and upsampling layers connected in series; wherein, RepConvBlock internally contains residual depthwise separable convolutions, dilated reparameterized blocks, batch normalization layers, and channel attention units; this unit is used to extract adaptive aligned feature maps and generate global context features consistent with the input resolution; S2.3.2: Construct a local fine-grained perception unit; it consists of context-mixed dynamic convolutional blocks; its input end has a splicing and fusion layer, which is used to splice and fuse adaptive aligned feature maps with global context features; the unit has a query vector generator and a key vector generator, which generate query vectors and key vectors from the spliced ​​and fused features and global context features, respectively, and generate global association weights through dot product operation; S2.3.3: Constructing a dynamic convolution and enhancement unit; including a dynamic parameter projection layer, a grouped convolutional layer, a positional encoding stacking layer, and an SE gated unit; the dynamic parameter projection layer maps the global correlation weights to two sets of dynamic convolutional kernel parameters corresponding to small kernels and large kernels respectively; the grouped convolutional layer uses the dynamic convolutional kernel parameters to perform grouped convolution on the adaptively aligned feature maps; the positional encoding stacking layer contains an internal dilated reparameterization block, and adds the grouped convolution output to the local enhancement positional encoding generated by the dilated reparameterization block, and finally modulates the output of the local refinement feature through the SE gated unit; S2.3.4: Construct the feature output layer; The feature output layer is connected to the end of the local fine-grained perceptual unit. The local fine-grained features output by the stacked context-mixed dynamic convolutional blocks are batch normalized and convolutionally projected to obtain the final co-refined features.

7. The optical flow estimation method based on depth perception and global-local coordination as described in claim 3, characterized in that, Step S2.4 specifically includes: S2.4.1: Construct the feature decoding layer; The feature decoding layer is connected after the local fine sensing unit and is used to update the hidden state. Its output is connected to the optical flow prediction head and the upsampling weight prediction head, respectively. S2.4.2: Constructing the prediction head unit; The optical flow prediction head consists of two convolutional layers with different kernel sizes connected in series, used to output the optical flow update amount of the two channels and additional optical flow uncertainty parameters; The upsampling weight prediction head consists of convolutional layers, used to output upsampling mask weights for a set number of channels; S2.4.3: Construct a convex combination upsampling unit; the input end receives the optical flow update, the current optical flow estimation map and the upsampling mask weights, accumulates the optical flow update to the current optical flow estimation map, and uses the mask weights to perform a weighted convex combination calculation of the local spatial neighborhood of the accumulated low-resolution optical flow to achieve double upsampling.

8. The optical flow estimation method based on depth perception and global-local coordination as described in claim 6, characterized in that, The RepConvBlock in step S2.3.1 and the dilated reparameterization block in step S2.3.3 adopt a multi-branch parallel structure to enhance feature extraction capabilities during the training phase, and in the inference phase, the multi-branch parameters are equivalently fused into a single convolutional kernel to achieve structural reparameterization and accelerate inference.

9. The optical flow estimation method based on depth perception and global-local coordination as described in claim 1, characterized in that, Step S3 includes: S3.1: Set the network training parameters, including batch size, initial learning rate, maximum number of iterations, and number of iterations; use the AdamW optimizer to iteratively train the optical flow estimation network; S3.2: Input the images in the training set into the optical flow estimation network in batches, and iterate through the DFE module, AWM module, OFRM module and OFPM module in sequence multiple times to obtain the optical flow prediction results and uncertainty parameters output in each iteration; S3.3: Construct a hybrid loss function; For the optical flow prediction results output in each iteration, calculate the endpoint error between the optical flow prediction results and the true optical flow map, and construct a normalized flow loss by combining the uncertainty parameters. By weighting and summing the losses of each iteration according to the exponential decay weight, the prediction results closer to the final iteration are given a larger weight, and the final hybrid loss function value is obtained. S3.4: Based on the hybrid loss function value, the gradient is calculated and the network parameters are updated through the backpropagation algorithm; training stops when the maximum number of iterations is reached, and the trained optical flow estimation network is obtained. S3.5: Use the validation set to evaluate the performance of the model during the training process, and select the model parameters with the smallest upper endpoint error on the validation set as the final optical flow estimation network.

Images