Optical flow estimation method and system

Abstract

The invention discloses an optical flow estimation method and system, and belongs to the field of computer vision and artificial intelligence. Firstly, two input frames of images are normalized, multi-scale features are extracted through a deep neural network, and then feature correlation is calculated to construct motion matching information. In the training stage, the robustness of a complex scene is improved through adaptive noise disturbance modeling, in the reasoning stage, an optical flow field is rapidly reconstructed through efficient diffusion sampling, in the diffusion step, features are enhanced through a gating unit, and finally high-precision optical flow is output through recursive optimization and up-sampling. The algorithm can be deployed in a specific hardware device, provides a multi-language standardized interface, and is suitable for scenes with requirements for real-time performance and precision, such as video analysis and automatic driving.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the field of computer vision and artificial intelligence, in particular to a method and system for optical flow estimation. BACKGROUND

[0002] Optical flow estimation is a core task in the field of computer vision, and its core goal is to estimate the motion vector (optical flow field) between adjacent two frames of image in a video sequence, which contains the motion direction and amplitude information of the pixel. High-precision optical flow estimation is a key prerequisite for high reliability in downstream tasks such as motion prediction of target tracking, obstacle dynamic detection of automatic driving, and environment perception of robot vision.

[0003] Current mainstream optical flow estimation methods are mainly based on optimization methods, convolutional neural networks (CNN) or Transformer architecture, but there are some key problems in practical application. The existing algorithms are prone to optical flow estimation deviation when dealing with large displacement (such as fast-moving targets), complex motion (such as rotation + translation composite motion) or dynamic scenes (such as sudden changes in light, occlusions), and the fundamental reason is that the physical characteristics (direction and amplitude correlation) of optical flow are not fully utilized to construct the training target, resulting in limited model generalization ability; in recent years, the diffusion probability model (DPM) has been introduced into the field of optical flow estimation due to its excellent generation modeling capability, but it requires a large number of sampling steps (traditional DDIM sampler requires 50-100 steps) in the reasoning stage, resulting in slow reasoning speed and difficulty in meeting the real-time requirements (usually requiring frame rate ≥ 15 fps) of end-side devices (such as automatic driving vehicle terminals and robot edge computing modules); the existing methods do not effectively combine the timing information of diffusion steps and the importance differences of feature channels in the feature fusion process, and cannot dynamically adjust the feature expression, resulting in weak adaptability of the model to different motion patterns, especially in multi-scale feature fusion, which easily loses key motion information.

[0004] In addition, the existing noise disturbance generation method of optical flow estimation method has significant defects: traditional methods generate noise using standard normal distribution, without considering the direction and amplitude characteristics of optical flow, resulting in large deviation between training target and real motion distribution, not only reducing the model convergence speed, but also leading to the decline of the estimation accuracy of the model for non-uniform motion in real scenes. At the same time, the existing algorithms lack targeted optimization at the hardware deployment level, making it difficult to balance the "high precision" and "high real-time" requirements on edge devices (usually ≤20Tops), limiting its large-scale application in industrial scenarios.

[0005] Therefore, it is necessary to develop an optical flow estimation method and system that can balance "high precision (adapt to complex motion)", "high efficiency (fast reasoning)", and "easy deployment (adapt to edge hardware)", which is the key to solving the current technical bottleneck. SUMMARY

[0006] This application provides an optical flow estimation method and system. The technical solution is as follows: According to one aspect of this application, an optical flow estimation method is provided, the method comprising: The two input images are normalized and multi-scale feature extraction is performed to obtain multi-level feature maps with different spatial resolutions. Perform dot product operations on the multi-level feature maps of the two frames to construct multi-level correlation volume information for capturing motion matching relationships between pixels; During the training phase, an adaptive noise perturbation target generation mechanism is used for diffusion modeling. Optical flow diffusion sampling is performed based on DPM-Solver during the inference phase; In each diffusion step, GatedEE gated units are used to embed and enhance the features; The low-resolution optical flow field is progressively optimized by a recursive update module, and the high-resolution optical flow field is reconstructed by a multi-scale upsampling module, outputting the final high-precision optical flow estimation result.

[0007] Optionally, the normalization process involves mapping pixel values ​​from the range [0, 255] to the range [-1, 1].

[0008] Optionally, the training phase employs an adaptive noise perturbation target generation mechanism for diffusion modeling, including: The direction angle and amplitude of the optical flow are calculated. The direction angle is obtained by arctan2(optical flow y component, optical flow x component), and the amplitude is calculated by sqrt(optical flow x component² + optical flow y component²). Generate a horizontal noise random normal distribution × (0.08 + 0.02 × sigmoid (amplitude)) and a vertical noise random normal distribution × (0.05 + 0.03 × sigmoid (amplitude)), so that the noise amplitude is adaptively adjusted with the optical flow amplitude; Calculate the horizontal weight cos(direction angle)² and the vertical weight sin(direction angle)², and weight the noise in the horizontal and vertical directions so that the noise mainly acts on the current main direction of motion; The weighted noise is combined with the normalized (real optical flow / feature map size) and scaled (normalized optical flow × 0.5) real optical flow to generate the diffusion model training target.

[0009] Optionally, the optical flow diffusion sampling based on DPM-Solver in the inference stage includes: Initialize parameters, determine the target shape based on the input feature shape, and generate initial random noise on the current device; Configure DPM-Solver parameters, set the number of training time steps to 1000, the initial beta value to 0.0001 and the ending beta value to 0.02, adopt the linear beta scheduling method, set the inference time steps from 4 to 20 and generate the time step sequence; Iterative sampling is performed. For each time step, a time step tensor input model is created to predict noise and internal optical flow. The noise and current coordinates are updated using the DPM-Solver high-order numerical solution method, and the optical flow results are collected. Extract the final results, obtain the network state, upsampling mask and final coordinates from the model prediction state, and output the optical flow sequence and final optical flow field.

[0010] Optionally, in each diffusion step, the feature embedding enhancement is performed using a GatedEE gated unit, including: The feature enhancement module sequentially performs group normalization, Gaussian error linear unit (GELU) activation, and two-dimensional convolution operations on the input features to output enhanced features. Through the channel gating module, the enhanced features are aggregated with global information through a two-dimensional adaptive average pooling layer, and after processing by two-dimensional convolution, GELU activation, and two-dimensional convolution, channel attention weights are generated through the Sigmoid activation function. The enhanced features are multiplied by the channel attention weights to obtain the channel gating features; A linear transformation (channel-gated feature × (scale + 1) + shift) is performed on the channel-gated features to generate temporal embedding features; The embedding enhancement module sequentially performs GELU activation and two-dimensional convolution operations on the temporal embedding features to output the embedding enhancement features. By embedding a gating module, GELU activation and two-dimensional convolution operations are sequentially performed on the temporal embedding features, and the embedding attention weights are generated by the Sigmoid activation function. The embedded enhancement features are multiplied by the embedded attention weights and then multiplied by a scaling factor (gamma) to obtain the gated embedded enhancement features; The input features are added to the gated embedding enhancement features to generate the final embedding enhancement residual features.

[0011] Optional, also includes: During the training phase, a multi-step recursive optimization strategy with noise perturbation is adopted. The loss between predicted optical flow and real optical flow is calculated, and the network parameters are updated using backpropagation until the model converges. During the inference phase, the DPM-Solver high-efficiency sampler described in claim 3 is used for fast inference, thereby separating the sampling strategies for the training and inference phases.

[0012] On the other hand, an optical flow estimation system is provided, including: The feature extraction module is used to perform normalization processing and multi-scale feature extraction on the two input frames of images, and output multi-level feature maps; The correlation calculation module is used to perform dot product operations on the multi-level feature maps of two frames of images to generate multi-level correlation volume information. The adaptive noise perturbation diffusion modeling module is used to generate adaptive, direction-weighted noise perturbations based on the optical flow direction and amplitude during the training phase, and to construct the diffusion model training target. The DPM-Solver diffusion sampling module is used to perform 4-20 steps of sampling during the inference phase using high-order numerical solutions to achieve efficient optical flow field reconstruction. The GatedEE gated embedding enhancement module is used to perform channel gating and temporal modulation on features during the diffusion process and output dynamically enhanced features. The recursive update module is used to progressively optimize and reconstruct the low-resolution optical flow field to a high-resolution version, and output the final optical flow result. The hardware platform module, built on the D-Robotics RDKX5 device, provides 10 TOPS of computing power to support algorithm execution. The interface module provides standardized calling interfaces supporting C / C++, Python, and Java, receiving image input and outputting optical flow maps.

[0013] In this embodiment, the two input frames are first normalized and multi-scale features are extracted using a deep neural network. Then, feature correlation is calculated to construct motion matching information. During the training phase, adaptive noise perturbation modeling is used to improve robustness to complex scenes. During the inference phase, efficient diffusion sampling is used to quickly reconstruct the optical flow field. In the diffusion step, gating units are used to enhance features. Finally, high-precision optical flow is output through recursive optimization and upsampling. The algorithm can be deployed on specific hardware devices, provides multi-language standardized interfaces, and is suitable for scenarios with real-time and accuracy requirements, such as video analysis and autonomous driving. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0015] Figure 1 This embodiment illustrates a model training and inference flowchart. Figure 2 This diagram illustrates a gated embedded enhancement GatedEE architecture provided in this embodiment. Detailed Implementation

[0016] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0017] In this article, "multiple" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0018] Example 1 An optical flow estimation method is provided, the method comprising: Step 101: Normalize and extract multi-scale features from the two input frames to obtain multi-level feature maps with different spatial resolutions.

[0019] The normalization process maps pixel values ​​from the range [0, 255] to the range [-1, 1]. The calculation formula is: normalized_image = 2 × (image / 255.0) - 1.0.

[0020] Multi-scale feature extraction is performed using deep neural networks (such as Twins-SVT-Large), outputting feature maps with at least two spatial resolutions, including a 1 / 8 resolution feature map (for capturing global motion information) and a 1 / 4 resolution feature map (for capturing local detailed motion information), providing multi-level contextual support for subsequent correlation calculations.

[0021] Step 102: Perform dot product operation on the multi-level feature maps of the two frames of images to construct multi-level correlation volume information for capturing motion matching relationships between pixels.

[0022] In practice, let the feature maps of the two frames be respectively (The shape is [batch, C, ...) , ])and (The shape is [batch, C, ...) , ]), where C is the number of feature channels, , and , These represent the height and width of the feature maps in the two frames, respectively.

[0023] right and The feature vector (with dimension C) of each pixel is subjected to a dot product operation to obtain a matrix reflecting pixel similarity. After further compressing the channel dimension, a multi-level correlation volume matching the resolution of the feature map is generated (e.g., a 1 / 8 resolution feature map corresponds to a 1 / 8 resolution correlation volume, and a 1 / 4 resolution feature map corresponds to a 1 / 4 resolution correlation volume). This volume directly reflects the motion correlation strength of pixels in two frames of images.

[0024] Step 103: During the training phase, an adaptive noise perturbation target generation mechanism is used for diffusion modeling.

[0025] In one possible implementation, step 103 includes the following.

[0026] S31, calculate the direction angle and amplitude of the optical flow. The direction angle is obtained by arctan2(optical flow y component, optical flow x component), and the amplitude is calculated by sqrt(optical flow x component² + optical flow y component²).

[0027] The optical flow x component corresponds to horizontal movement, and the optical flow y component corresponds to vertical movement. The direction angle range is [-π, π]. The larger the amplitude value, the farther the pixel moves.

[0028] S32 generates a horizontal noise random normal distribution × (0.08 + 0.02 × sigmoid(amplitude)) and a vertical noise random normal distribution × (0.05 + 0.03 × sigmoid(amplitude)), so that the noise amplitude is adaptively adjusted with the optical flow amplitude; the sigmoid function is used to map the amplitude value to the range [0,1], ensuring that the noise intensity in the large motion region (large amplitude value) is higher than that in the small motion region, thus improving the model's ability to learn large displacement motion.

[0029] S33, calculate the horizontal weight cos(direction angle)² and the vertical weight sin(direction angle)², and weight the noise in the horizontal and vertical directions so that the noise mainly acts on the current main direction of motion.

[0030] For example, when the main direction of optical flow is horizontal, the weight of the horizontal direction approaches 1, and the weight of the vertical direction approaches 0. The noise is mainly concentrated in the horizontal direction, reducing the interference of irrelevant directional noise on training.

[0031] S34 combines the weighted noise with the normalized (real optical flow / feature map size) and scaled (normalized optical flow × 0.5) real optical flow to generate the diffusion model training target.

[0032] The "True Optical Flow / Feature Map Size" is used to eliminate the influence of feature maps of different resolutions on the range of optical flow values, and "×0.5" is used to compress the range of optical flow values ​​to avoid numerical overflow during training.

[0033] S35, random sampling diffusion time step, with a time step value range of [0,1000]. By simulating training scenarios with different noise intensities during the diffusion process through random time steps, the generalization ability of the model can be further improved.

[0034] S36 performs a pruning operation on the generated noisy training target, limiting the numerical range to [-1,1], and then multiplies it by a scaling factor of 0.1 to ensure the numerical stability of the training target and avoid interference from extreme values ​​on the model parameter update.

[0035] Step 104: Optical flow diffusion sampling is performed based on DPM-Solver during the inference phase.

[0036] In one possible implementation, step 104 includes the following.

[0037] S41, initialize parameters, determine the target shape based on the input feature shape, and generate initial random noise on the current device.

[0038] Specifically, the target optical flow field shape ([batch,2,H,W], where "2" corresponds to the horizontal and vertical motion components respectively) is determined based on the input feature shape (e.g., [batch,C,H,W]), and initial random noise conforming to a standard normal distribution is generated on the current hardware device (e.g., the Sunrise5 chip of the D-RoboticsRDKX5 device).

[0039] S42, configure DPM-Solver parameters, set the training time steps to 1000, the initial beta value to 0.0001 and the ending beta value to 0.02, adopt the linear beta scheduling method, set the inference time steps from 4 to 20 and generate the time step sequence.

[0040] Set the inference time steps to 4-20 (which can be adjusted according to real-time requirements, such as 10-15 steps for autonomous driving scenarios and 8-12 steps for robot vision scenarios), and generate a time step sequence based on linear scheduling (e.g., when the inference steps are 10, the time step sequence can be set to [990,890,790,690,590,490,390,290,190,90]).

[0041] S43, iterative sampling: for each time step, create a time step tensor input model to predict noise and internal optical flow, update the noise and current coordinates through the DPM-Solver high-order numerical solution, and collect optical flow results.

[0042] Furthermore, iterative sampling is performed, and the following operations are performed for each time step t: S431, Create a time step tensor. The tensor has the shape [batch] and all values ​​are the current time step t. The data type is set to long integer. This is used to inform the model of the current diffusion stage. S432, input the time step tensor into the optical flow model to predict the current noise, internal optical flow, and model prediction state; where the internal optical flow is the intermediate optical flow result at this time step, which is used for subsequent result summarization and optimization; S433 updates the prediction noise using a higher-order numerical solution method of DPM-Solver (such as the third-order Runge-Kutta method) and dynamically adjusts the sampling step size to balance accuracy and efficiency. S434, adjust the current coordinates according to the updated noise (current coordinates = initial coordinates + updated noise) to achieve gradual restoration of the optical flow field; S435 adds the internal optical flow of the current time step to the optical flow results list for subsequent multi-scale fusion and optimization.

[0043] S44, extract the final result, obtain the network state, upsampling mask and final coordinates from the model prediction state, and output the optical flow sequence and final optical flow field.

[0044] Specifically, the final result is extracted, and the network state (for subsequent recursive optimization), upsampling mask (for boundary correction during multi-scale upsampling), and final coordinates are obtained from the model prediction state. An initial optical flow field is generated based on the final coordinates, and combined with the intermediate optical flow in the optical flow result list, the complete optical flow sequence and the final optical flow field are output.

[0045] Therefore, a multi-step recursive optimization strategy with noise perturbation is adopted during the training phase. This strategy calculates the loss between predicted optical flow and actual optical flow and updates the network parameters using backpropagation until the model converges. Specifically, the loss calculation uses a combination of "Endpoint Error (EPE) loss + Multi-scale loss". The multi-scale loss is 1 / 8, 1 / 4, and the original resolution optical flow EPE loss is summed with weights (0.2, 0.3, 0.5). The optimizer is AdamW, with a learning rate of 1e-4 and a weight decay coefficient of 1e-5. The gradients of all model parameters (feature extraction network, correlation calculation module, GatedEE module, etc.) are calculated through backpropagation, and the parameters are iteratively updated. The training termination condition is that the EPE loss on the validation set (such as 10% of the Sintel training set) does not decrease for 10 consecutive rounds. At this point, the converged model parameters are saved for the inference phase.

[0046] During the inference phase, the aforementioned DPM-Solver high-efficiency sampler is used for fast inference, achieving separation of sampling strategies between the training and inference phases, thus balancing training accuracy and inference efficiency.

[0047] Step 105: In each diffusion step, the features are embedded and enhanced using a GatedEE gated unit.

[0048] In one possible implementation, step 105 includes the following.

[0049] S51, through the feature enhancement module, sequentially performs group normalization, Gaussian error linear unit (GELU) activation, and two-dimensional convolution operations on the input features, and outputs enhanced features.

[0050] Specifically, the input features (of shape [batch, C, H, W]) are processed sequentially as follows: group normalization (num_groups=C / / 32, ensuring uniform grouping across channels), Gaussian error linear unit (GELU) activation (introducing nonlinearity to mitigate gradient vanishing), and 2D convolution operation (kernel_size=3, padding=1, ensuring the number of output channels matches the number of input channels), resulting in enhanced output features. .

[0051] S52, through the channel gating module, summarizes global information of the enhanced features through a two-dimensional adaptive average pooling layer, and after processing by two-dimensional convolution, GELU activation, and two-dimensional convolution, generates channel attention weights through the Sigmoid activation function.

[0052] Enhanced features are controlled through the channel gating module. Perform the following operations: S521 employs a two-dimensional adaptive average pooling layer (output_size=1) for... Global information is aggregated to obtain global features of shape [batch, C, 1, 1]. S522, perform two-dimensional convolution (kernel_size=3, padding=1, number of output channels=C / / 2), GELU activation, and two-dimensional convolution (kernel_size=1, number of output channels=C) on the global features in sequence. S523, the above results are processed using the Sigmoid activation function to generate channel attention weights. (with a shape of [batch, C, 1, 1]), the weights reflect the importance of each feature channel.

[0053] S53 multiplies the enhanced features with the channel attention weights to obtain the channel gating features.

[0054] Enhance features With channel attention weights Element-wise multiplication is performed to obtain the channel gating features. This enables the enhancement of important feature channels and the suppression of irrelevant channels.

[0055] S54 performs a linear transformation on the channel-gated features using the scale vector and shift vector (channel-gated feature × (scale + 1) + shift) to generate temporal embedding features.

[0056] Channel gating features Temporal embedding features are generated by performing a linear transformation on the scale vector and the shift vector. ; where scale and shift are learnable parameters, both with dimension C, used to introduce temporal information of the diffusion step and improve the model's adaptability to changes in motion temporal sequence.

[0057] S55, through the embedding enhancement module, sequentially performs GELU activation and two-dimensional convolution operations on the temporal embedding features, and outputs the embedding enhancement features.

[0058] By embedding enhancement modules, temporal embedding features are enhanced. GELU activation and two-dimensional convolution operations are performed sequentially (kernel_size=3, padding=1, number of output channels=C), and the embedded enhanced feature x_feature is output.

[0059] S56, through the embedded gating module, sequentially performs GELU activation and two-dimensional convolution operations on the temporal embedded features, and generates embedded attention weights through the Sigmoid activation function.

[0060] By embedding a gating module, temporal embedding features are... GELU activation and two-dimensional convolution operations are performed sequentially (kernel_size=3, padding=1, number of output channels=C). Then, the embedding attention weight x_gate (shape [batch,C,H,W]) is generated through the Sigmoid activation function. This weight is used to control the weighting degree of the embedding enhancement features.

[0061] S57 multiplies the embedding enhancement feature with the embedding attention weight and then multiplies it by a scaling factor (gamma) to obtain the gated embedding enhancement feature.

[0062] The element-wise multiplication of the embedding enhancement feature x_feature and the embedding attention weight x_gate is performed, and then multiplied by a scaling factor (gamma, with a value of 0.1, used to balance the intensity of feature enhancement) to obtain the gated embedding enhancement feature x_ee.

[0063] S58 adds the input features to the gated embedding enhancement features to generate the final embedding enhancement residual features.

[0064] The input features are added element-wise to the gated embedding enhancement features x_ee to generate the final embedding enhancement residual features x_output. This operation enhances key features while preserving the original feature information and avoids the gradient vanishing problem.

[0065] Step 106: The low-resolution optical flow field is gradually optimized by the recursive update module, and the high-resolution optical flow field is reconstructed by the multi-scale upsampling module, and the final high-precision optical flow estimation result is output.

[0066] In one possible implementation, step 106 is performed as follows.

[0067] Recursive optimization involves inputting the embedded enhanced residual features output by GatedEE into the recursive update module and setting three iterations for optimization.

[0068] In the first iteration, the initial optical flow field at 1 / 8 resolution is optimized based on the 1 / 8 resolution correlation volume to correct global motion deviations. In the second iteration, the optimized 1 / 8 resolution optical flow field is upsampled to 1 / 4 resolution, and the local motion deviation is further corrected by combining the 1 / 4 resolution correlated volume. The third iteration further optimized the 1 / 4 resolution optical flow field, improving the detail accuracy of the optical flow. Multi-scale upsampling is performed by using a transposed convolutional layer (kernel_size=4, stride=2, padding=1) to upsample the optimized 1 / 4 resolution optical flow field, gradually restoring it to the original image resolution (e.g., when the input image is 1920×1080, the final optical flow field resolution is 1920×1080). During the upsampling process, an upsampling mask (from S44 in step 104) is introduced to correct the motion information of boundary pixels and avoid the loss of details caused by interpolation. The output format of the final high-resolution optical flow field is converted to produce an optical flow map containing horizontal and vertical motion information. The pixel values ​​of the optical flow map correspond to the magnitude of the motion vector, which can be directly used for downstream tasks such as video analysis and obstacle detection in autonomous driving.

[0069] Finally, the above algorithm is deployed on hardware devices to achieve efficient operation and standardized invocation.

[0070] In one example, the hardware device is a D-Robotics RDKX5, which is equipped with a Sunrise5 intelligent computing chip and has 10 TOPS computing power. The deployment process includes the following.

[0071] Model quantization uses INT8 quantization technology to process the converged training model, converting 32-bit floating-point parameters into 8-bit integers, reducing model size by more than 75% and improving computational efficiency by 4 times. Hardware adaptation: For the multi-core parallel architecture and dedicated convolution acceleration unit of the Sunrise5 chip, the core operators of the model (such as convolution, pooling, and GELU activation) are reconstructed to ensure compatibility between the operators and the hardware acceleration unit. The interface is encapsulated and provides a standardized calling interface that supports three programming languages: C / C++, Python, and Java. The interface input consists of two RGB images (resolution supports 640×480 to 1920×1080) and the output is a single-channel optical flow map. Performance optimization: Through the chip's hardware acceleration engine and parallel computing technology, the inference speed is optimized to 25fps (1920×1080 resolution) to meet the real-time requirements of scenarios such as autonomous driving and robot vision.

[0072] Example 2 An optical flow estimation system, comprising: The feature extraction module is used to perform normalization processing and multi-scale feature extraction on the two input frames of images, and output multi-level feature maps; The correlation calculation module is used to perform dot product operations on the multi-level feature maps of two frames of images to generate multi-level correlation volume information. The adaptive noise perturbation diffusion modeling module is used to generate adaptive, direction-weighted noise perturbations based on the optical flow direction and amplitude during the training phase, and to construct the diffusion model training target. The DPM-Solver high-efficiency diffusion sampling module is used to perform 4-20 steps of sampling during the inference stage using high-order numerical solutions to achieve efficient optical flow field reconstruction. The GatedEE gated embedding enhancement module is used to perform channel gating and temporal modulation on features during the diffusion process and output dynamically enhanced features. The recursive update module is used to progressively optimize and reconstruct the low-resolution optical flow field to a high-resolution version, and output the final optical flow result. The hardware platform module is built on the D-Robotics RDKX5 device (equipped with a Sunrise5 chip) and provides 10 TOPS of computing power to support algorithm execution. The interface module provides standardized calling interfaces supporting C / C++, Python, and Java, receiving image input and outputting optical flow maps.

[0073] The following description, in conjunction with various embodiments and accompanying drawings, provides further details.

[0074] like Figure 1 As shown, Figure 1 The complete process of this optical flow estimation method in the two core stages of training and inference is clearly presented, as follows.

[0075] In terms of overall structure,Figure 1 It is divided into two parts: (a) model training and (b) model inference, which are displayed side by side to facilitate a direct comparison of the process differences and connections between the two stages. The core of both stages revolves around the technical line of "multi-scale feature extraction - correlation calculation - feature enhancement - optical flow optimization" to ensure the consistency of technical logic between training and inference.

[0076] For (a) the model training part, the process begins with the input of the "optical flow image dataset" and "two images". First, the images are preprocessed through the "multi-scale feature extraction" module to obtain feature maps of different resolutions, laying the foundation for subsequent motion information capture. Then, the "correlation volume calculation" module constructs the motion matching relationship between pixels to form multi-level correlation data. Subsequently, the core "adaptive noise perturbation target generation and diffusion modeling" step is entered. This step generates physically reasonable noise based on the direction and amplitude characteristics of optical flow, making the training target more consistent with the real motion distribution. Then, the features are dynamically optimized through "gated embedding enhanced feature modulation (GatedEE)" to strengthen key motion information. Afterward, the optical flow field is gradually refined through "recursive optimization and multi-scale upsampling". At the same time, the error between the predicted optical flow and the real optical flow is calculated by combining "loss calculation and backpropagation". The gradient information is fed back to the "parameter update" step to continuously adjust the model parameters and finally complete the training iteration of the "optical flow model" until the model converges.

[0077] For the model inference part (b), the input is the two images to be processed. It also goes through the "multi-scale feature extraction" and "correlation volume calculation" modules, sharing the same feature preprocessing logic as the training stage. The key difference is that the noise modeling step in the training stage is replaced. The "efficient optical flow diffusion sampling algorithm based on DPM-Solver" is adopted to quickly reconstruct the optical flow field through a limited number of sampling steps of 4-20, which greatly improves the inference efficiency. Subsequently, it also goes through the "gated embedding enhanced feature modulation (GatedEE)" and "recursive optimization and multi-scale upsampling" modules to refine the optical flow field. Finally, it directly outputs a "high-precision optical flow map", which meets the dual requirements of real-time performance and accuracy in practical applications. The whole process does not require loss of computation and parameter updates, and focuses on generating efficient optical flow estimation results.

[0078] like Figure 2 As shown, Figure 2 The internal structure and feature processing flow of the GatedEE gated unit were dissected in detail. From input features to final output enhanced features, each module has a clear function and works in concert, as detailed below.

[0079] The process begins with "Input Feature x0" and first enters the "Feature Enhancement" module. This module performs preliminary processing on the input features through a series of operations: "Group Normalization - Gaussian Error Linear Unit (GELU) - 2D Convolutional Layer - Group Normalization". This effectively extracts and enhances key feature information, and finally outputs "Enhanced Feature x1", laying a good feature foundation for subsequent channel gating.

[0080] The "enhanced feature x1" flows to two branches simultaneously. One branch enters the "channel gating" module: first, the enhanced feature is aggregated globally through a "two-dimensional adaptive average pooling layer" to compress the spatial dimension and capture global feature correlations; then, it undergoes further feature transformation through a "two-dimensional convolutional layer-GELU activation-two-dimensional convolutional layer"; finally, the "channel attention weight x2" is generated through the "Sigmoid activation function". This weight can accurately reflect the importance of each feature channel; then, the "enhanced feature x1" and the "channel attention weight x2" are multiplied element-wise to obtain the "channel gating feature x3", which strengthens important feature channels and suppresses irrelevant channels.

[0081] The “channel gating feature x3” enters the “temporal embedding feature calculation” stage. By performing a linear transformation with the learnable “scale vector” and “shift vector” (the calculation formula is x4=x3×(scale+1)+shift), the “temporal embedding feature x4” is generated, which successfully introduces the temporal information of the diffusion step and improves the model’s ability to adapt to changes in motion temporal sequence.

[0082] The "temporal embedding feature x4" is further divided into two branches: one branch enters the "embedding enhancement" module, which outputs "embedding enhancement feature x_feature" after processing by the "GELU activation-2D convolutional layer"; the other branch enters the "embedding gating" module, which is also processed by the "GELU activation-2D convolutional layer" and then generates "embedding attention weight x_gate" through the "Sigmoid activation function". This weight is used to control the weighting degree of the embedding enhancement feature. Then, the "embedding enhancement feature x_feature" and the "embedding attention weight x_gate" are multiplied element-wise and multiplied by the scaling factor (gamma) to obtain the "gated embedding enhancement feature x_ee", realizing the dynamic adjustment of the feature.

[0083] Finally, the initial "input feature x0" and "gated embedding enhancement feature x_ee" are added element-wise to generate "embedding enhancement residual feature x_output". This operation not only preserves the original feature information, but also further enhances the key features, effectively avoids the gradient vanishing problem, and provides high-quality feature support for subsequent optical flow field optimization.

[0084] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0085] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. An optical flow estimation method, characterized in that, The method includes: The two input images are normalized and multi-scale feature extraction is performed to obtain multi-level feature maps with different spatial resolutions. Perform dot product operations on the multi-level feature maps of the two frames to construct multi-level correlation volume information for capturing motion matching relationships between pixels; During the training phase, an adaptive noise perturbation target generation mechanism is used for diffusion modeling. Optical flow diffusion sampling is performed based on DPM-Solver during the inference phase; In each diffusion step, GatedEE gated units are used to embed and enhance the features; The low-resolution optical flow field is progressively optimized by a recursive update module, and the high-resolution optical flow field is reconstructed by a multi-scale upsampling module, outputting the final high-precision optical flow estimation result.

2. The method according to claim 1, characterized in that, The normalization process involves mapping pixel values ​​from the range [0, 255] to the range [-1, 1].

3. The method according to claim 1, characterized in that, The training phase employs an adaptive noise perturbation target generation mechanism for diffusion modeling, including: The direction angle and amplitude of the optical flow are calculated. The direction angle is obtained by arctan2(optical flow y component, optical flow x component), and the amplitude is calculated by sqrt(optical flow x component² + optical flow y component²). Generate a horizontal noise random normal distribution × (0.08 + 0.02 × sigmoid (amplitude)) and a vertical noise random normal distribution × (0.05 + 0.03 × sigmoid (amplitude)), so that the noise amplitude is adaptively adjusted with the optical flow amplitude; Calculate the horizontal weight cos(direction angle)² and the vertical weight sin(direction angle)², and weight the noise in the horizontal and vertical directions so that the noise mainly acts on the current main direction of motion; The weighted noise is combined with the normalized (real optical flow / feature map size) and scaled (normalized optical flow × 0.5) real optical flow to generate the diffusion model training target.

4. The method according to claim 1, characterized in that, The inference phase, based on DPM-Solver, performs optical flow diffusion sampling, including: Initialize parameters, determine the target shape based on the input feature shape, and generate initial random noise on the current device; Configure DPM-Solver parameters, set the number of training time steps to 1000, the initial beta value to 0.0001 and the ending beta value to 0.02, adopt the linear beta scheduling method, set the inference time steps from 4 to 20 and generate the time step sequence; Iterative sampling is performed. For each time step, a time step tensor input model is created to predict noise and internal optical flow. The noise and current coordinates are updated using the DPM-Solver high-order numerical solution method, and the optical flow results are collected. Extract the final results, obtain the network state, upsampling mask and final coordinates from the model prediction state, and output the optical flow sequence and final optical flow field.

5. The method according to claim 1, characterized in that, In each diffusion step, the features are embedded and enhanced using GatedEE gated units, including: The feature enhancement module sequentially performs group normalization, Gaussian error linear unit (GELU) activation, and two-dimensional convolution operations on the input features to output enhanced features. Through the channel gating module, the enhanced features are aggregated with global information through a two-dimensional adaptive average pooling layer, and after processing by two-dimensional convolution, GELU activation, and two-dimensional convolution, channel attention weights are generated through the Sigmoid activation function. The enhanced features are multiplied by the channel attention weights to obtain the channel gating features; A linear transformation (channel-gated feature × (scale + 1) + shift) is performed on the channel-gated features to generate temporal embedding features; The embedding enhancement module sequentially performs GELU activation and two-dimensional convolution operations on the temporal embedding features to output the embedding enhancement features. By embedding a gating module, GELU activation and two-dimensional convolution operations are sequentially performed on the temporal embedding features, and the embedding attention weights are generated by the Sigmoid activation function. The embedded enhancement features are multiplied by the embedded attention weights and then multiplied by a scaling factor (gamma) to obtain the gated embedded enhancement features; The input features are added to the gated embedding enhancement features to generate the final embedding enhancement residual features.

6. The method according to any one of claims 1 to 5, characterized in that, Also includes: During the training phase, a multi-step recursive optimization strategy with noise perturbation is adopted. The loss between predicted optical flow and real optical flow is calculated, and the network parameters are updated using backpropagation until the model converges. During the inference phase, the DPM-Solver high-efficiency sampler described in claim 3 is used for fast inference, thereby separating the sampling strategies for the training and inference phases.

7. An optical flow estimation system implementing any one of claims 1 to 6, characterized in that, include: The feature extraction module is used to perform normalization processing and multi-scale feature extraction on the two input frames of images, and output multi-level feature maps; The correlation calculation module is used to perform dot product operations on the multi-level feature maps of two frames of images to generate multi-level correlation volume information. The adaptive noise perturbation diffusion modeling module is used to generate adaptive, direction-weighted noise perturbations based on the optical flow direction and amplitude during the training phase, and to construct the diffusion model training target. The DPM-Solver diffusion sampling module is used to perform 4-20 steps of sampling during the inference phase using high-order numerical solutions to achieve efficient optical flow field reconstruction. The GatedEE gated embedding enhancement module is used to perform channel gating and temporal modulation on features during the diffusion process and output dynamically enhanced features. The recursive update module is used to progressively optimize and reconstruct the low-resolution optical flow field to a high-resolution version, and output the final optical flow result. The hardware platform module, built on the D-Robotics RDKX5 device, provides 10 TOPS of computing power to support algorithm execution. The interface module provides standardized calling interfaces supporting C / C++, Python, and Java, receiving image input and outputting optical flow maps.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which is loaded and executed by a processor to implement the optical flow estimation method as described in any one of claims 1 to 6.

Images