A method and apparatus for efficient optical flow estimation based on mamba

Abstract

The application discloses a kind of high-efficiency optical flow estimation method and device based on Mamba, its method includes: normalizing and size alignment to adjacent two frames of images, and extracting dense features of fixed down-sampling rate by shared weight convolutional encoder;Two frame features are sent into multi-level feature enhancement module, by frame modeling unit and cross-frame interaction unit cascade and cooperate channel reorganization and residual correction, obtain enhanced features;Four-dimensional cost volume is constructed on low resolution and is normalized along target coordinate dimension, corresponding coordinates are obtained by probability to target coordinate grid weighted, and difference is made with source coordinates, obtain initial optical flow;Initial optical flow and context and local correlation are guided by attention spatial fusion, send into autoregressive refinement module based on difference Mamba by small amount fixed step iteration update, and by convex combination up-sampling, restore to target resolution, output final optical flow.The application can accurately estimate optical flow field under low complexity and low latency.

Description

TECHNICAL FIELD

[0001] The application relates to the technical field of computer vision, and develops an efficient optical flow estimation method and device based on Mamba. BACKGROUND

[0002] Optical flow estimation is a basic problem in computer vision, aiming to describe scene motion information by analyzing pixel displacement between consecutive frames, and has important application value in video monitoring, moving target detection, automatic driving and robot navigation.

[0003] Traditional methods are mostly based on the assumptions of constant brightness and small displacement, and obtain the optical flow field through energy function optimization. Such methods are theoretically complete, but are often insufficient in robustness in fast motion, significant light changes or weak texture areas, and have large computational overhead, making it difficult to meet the real-time needs of engineering.

[0004] Deep learning has promoted the development of end-to-end optical flow methods. Convolutional neural network (CNN) based schemes are superior in efficiency, but are limited by the local receptive field of convolution; many methods rely on multiple iterations of refinement to achieve accuracy, which increases the number of parameters and time delay, and still has performance bottlenecks in large displacement and complex motion scenes. In contrast, the Transformer uses self-attention to achieve global modeling, which can better handle large displacement and complex motion, but its quadratic computational complexity results in slow inference speed, bloated models, high resource occupation, and limited deployment.

[0005] In summary, while maintaining estimation accuracy, reducing inference delay and computing power, and balancing global information modeling and efficient computation, are key contradictions that need to be solved in current optical flow research and application. Therefore, there is an urgent need for an efficient optical flow estimation scheme that can balance global dependence representation and linear / quasi-linear complexity to adapt to high-resolution, long-sequence and resource-constrained real application scenarios. SUMMARY

[0006] The present application overcomes the above-mentioned shortcomings of the prior art, and provides an efficient optical flow estimation method and device based on Mamba, which improves the high computational complexity and long inference delay of existing methods, significantly reduces the computational complexity and improves the inference speed while maintaining or even improving the accuracy.

[0007] To solve the above technical problems, the present application provides the following technical solutions:

[0008] The first aspect of the present application provides an efficient optical flow estimation method based on Mamba, comprising the following steps:

[0009] Collecting two adjacent frames of images to be processed and performing normalization and size alignment processing to form an input image pair with uniform resolution and dynamic range;

[0010] The image pairs are respectively subjected to feature extraction, and a convolutional encoder with shared weights outputs single-scale high-dimensional dense feature maps at a fixed down-sampling rate to balance between representation capability and computational complexity.

[0011] The two-frame dense features are input into a multi-level feature enhancement module, long-range context is aggregated within the frame first, then interactive fusion is performed in the cross-frame dimension, and a spatial neighborhood state fusion mechanism is introduced in the state updating process, so that the feature representation can combine sequence dependence and two-dimensional spatial context at the same time, and the multi-layer perception (MLP) channel reorganization and residual correction are combined to obtain global and cross-frame information supporting matching and estimation with linear or near-linear time complexity.

[0012] A four-dimensional cost volume is constructed at the low-resolution feature level to represent the correspondence relationship from the source pixel to the target pixel, softmax normalization is performed in the target coordinate dimension to obtain a matching probability distribution, and the two-dimensional coordinate grid of the target image is weighted and averaged based on the distribution to obtain the corresponding coordinates, and then the initial optical flow is obtained by subtracting the source coordinates, so that a high-quality initial value is obtained by one-time global matching, and the number of subsequent iterations and the total amount of calculation are reduced.

[0013] The initial optical flow and the local correlation of the target feature are jointly encoded into motion representation and sent into an attention-guided aggregator together with the context feature for spatial adaptive weighted fusion to highlight the dynamically relevant areas and suppress irrelevant interference, and aggregated features for refinement are obtained.

[0014] The aggregated features are input into an autoregressive refinement module based on differential Mamba, and the optical flow increment is gradually output according to a predetermined small number of fixed iteration steps and is superimposed on the current estimate to obtain a more accurate optical flow field.

[0015] The refined low-resolution optical flow is restored to the target resolution by convex combination up-sampling, and the final result reconstruction and boundary consistency processing are completed in combination with the local continuity constraint.

[0016] The final optical flow result is output for motion analysis, video understanding or other downstream visual tasks.

[0017] The feature extraction stage adopts a shallow residual network with shared weights as an encoder for two frames of images, and outputs single-scale high-dimensional features at a fixed down-sampling rate to balance between representation capability and computational complexity.

[0018] The encoder is composed of stacked basic residual units, and normalization and nonlinear activation are combined to stabilize training and inference.

[0019] To facilitate transfer learning and multi-channel input, the encoder supports loading general pre-trained weights and adapting the first layer weights.

[0020] Channel alignment and light fusion are set at the tail of the encoder to form a unified dimensional feature representation for subsequent enhancement and matching stages.

[0021] The multi-level feature enhancement module is composed of a plurality of intra-frame Mamba units and cross-frame Mamba units in cascade, and a light MLP is introduced after each layer to perform channel reorganization and residual correction, thereby enhancing the discriminability without significantly increasing the constant term overhead.

[0022] Global matching is realized in a global correlation manner on low-resolution features, and a global correlation matrix is calculated after the spatial dimension of the two frames is flattened, and is scaled by the square root of the number of channels, thereby obtaining a cost volume representing the one-to-one correspondence between source pixels and target pixels;

[0023] The cost volume is subjected to softmax normalization in the target coordinate dimension to obtain a matching probability distribution, and the corresponding coordinates are obtained by weighted averaging of the target coordinate grid based on the distribution.

[0024] The corresponding coordinates and the source coordinates are subtracted to obtain the initial optical flow, and if necessary, bidirectional optical flow is output by simultaneously calculating the forward and reverse correlations.

[0025] The autoregressive refinement module uses a fixed small number of iterations to control the inference delay and the amount of calculation, and the number of iterations is set as a predetermined constant during deployment, and is updated by accumulating the optical flow increment output by the differential Mamba-based decoding unit at each step.

[0026] The up-sampling is realized by convex combination of the neighborhood optical flow through the learned local weight, and the continuity reconstruction of sub-pixel accuracy is realized.

[0027] The training adopts an end-to-end joint optimization strategy to simultaneously improve the collaborative effect of the four stages of encoding, enhancement, matching and refinement, and data enhancement includes brightness and contrast disturbance, random cropping and pyramid scaling to enhance the generalization ability to large displacement, weak texture and repetitive texture scenes, and the loss design includes a reprojection consistency term, an occlusion and boundary robustness term and a smoothing regularization term.

[0028] The second aspect of the present application relates to a Mamba-based efficient optical flow estimation device, comprising a memory and one or more processors, the memory storing executable code, and the one or more processors executing the executable code to implement a Mamba-based efficient optical flow estimation method of the present application.

[0029] The third aspect of the present application relates to a computer-readable storage medium having a program stored thereon, which is executed by a processor to implement a Mamba-based efficient optical flow estimation method of the present application.

[0030] After the normalization and size alignment of the two adjacent frames of images are completed, the application extracts single-scale high-dimensional features with a fixed down-sampling rate through a shared weight convolutional encoder; the two frame features are input into a multi-level feature enhancement module for intra-frame and cross-frame information aggregation, and the intra-frame modeling unit and the cross-frame interaction unit are cascaded and cooperated with channel reorganization and residual correction to obtain enhanced features; a four-dimensional cost volume is constructed on a low resolution and is normalized along the target coordinate dimension, the corresponding coordinates are obtained by weighting the target coordinate grid according to the probability, and the difference between the source coordinates and the corresponding coordinates is calculated to perform probabilistic global matching to obtain the initial optical flow; the initial optical flow, the context features and the attention-guided spatial fusion of the local correlation are sent into an autoregressive refinement module based on the differential Mamba for iterative refinement, and are restored to the target resolution through convex combination up-sampling to output the final optical flow result, thereby realizing low complexity, high throughput and low latency on the premise of maintaining the existing accuracy.

[0031] The innovation of the application is:

[0032] (1) The Mamba state space model is introduced into the optical flow estimation framework, which breaks through the local receptive field limitation of traditional convolutional neural networks and the secondary complexity bottleneck of Transformer, realizes global dependence modeling under nearly linear complexity, and significantly reduces the consumption of computing resources and the inference time delay.

[0033] (2) A spatial neighborhood state fusion mechanism is proposed in the state recursion process of the intra-frame and cross-frame Mamba units, which improves the consistency and discriminability of the representation under large displacement, illumination change and occluded areas.

[0034] (3) An autoregressive refinement structure based on the differential Mamba is designed, which can highlight significant motion patterns and suppress redundant noise, effectively alleviate the state diffusion and numerical instability problems in the long sequence recursion process, and realize high-precision optical flow refinement in a small number of iteration steps.

[0035] (4) The attention mechanism is used to weight and aggregate the motion features, context features and historical states, realize spatial adaptive information fusion, highlight dynamic related areas and suppress irrelevant interference.

[0036] The working principle of the application is:

[0037] (1) The Mamba state space model establishes long-range dependencies under linear complexity by selective scanning and recursive updating, so that each position can combine input features and hidden states when updating, thereby ensuring that the optical flow estimation can perceive the global and maintain low computational overhead.

[0038] (2) The spatial neighborhood state fusion mechanism combines the current position hidden state and the state of the two-dimensional neighborhood in a weighted manner during state updating, so that the updating not only depends on the time sequence recursion, but also combines the local context. This joint modeling method can reduce matching ambiguity and enhance feature stability in large displacement and occlusion scenarios.

[0039] (3) The differential Mamba autoregressive refinement calculates two paths of states in parallel and performs differential fusion in each iteration to highlight significant motion patterns and suppress redundant noise. By gradually superimposing the flow increments, the model can converge to a higher precision result in a small number of fixed iteration steps.

[0040] (4) The attention mechanism generates weights for motion features, context features, and historical states pixel by pixel in the refinement stage, and then performs weighted summation after input dimension normalization, thereby adaptively highlighting motion-related regions, reducing irrelevant interference, and providing stable input for subsequent iterations.

[0041] The advantages of the present application are:

[0042] (1) An efficient optical flow estimation method based on Mamba is proposed, which can complete optical flow estimation with linear or near-linear complexity while maintaining global dependency modeling capability, significantly reducing computational resource consumption and inference time delay.

[0043] (2) Through multi-level feature enhancement of intra-frame and cross-frame Mamba units, and the spatial neighborhood state fusion mechanism, the estimation accuracy and robustness in large displacement, weak texture, illumination change, and occlusion scenarios are effectively improved.

[0044] (3) The autoregressive refinement module based on differential Mamba combined with the attention guided aggregator can complete optical flow refinement in a small number of iteration steps, balancing high precision and high real-time performance.

[0045] (4) The method has good generality and scalability, supports pre-trained weight loading and diversified data augmentation, and can be widely applied to visual task scenarios such as motion analysis, video understanding, autonomous driving, and robot navigation. BRIEF DESCRIPTION OF DRAWINGS

[0046] Figure 1 is the overall flowchart of the method of the present application.

[0047] Figure 2 is the overall architecture of the model of the method of the present application.

[0048] Figure 3 is a structural schematic diagram of multi-level feature enhancement of the present application.

[0049] Figure 4 is an iteration process schematic diagram of the autoregressive refinement module of the present application.

[0050] Figure 5 is a schematic diagram of the device of the present application. DETAILED DESCRIPTION

[0051] In order to make the technical problems, technical solutions and advantages of the present application clearer, a kind of high-efficiency end-to-end optical flow estimation method based on Mamba will be described in detail below with reference to the drawings.

[0052] Embodiment 1

[0053] In order to overcome the shortcomings of the existing optical flow estimation method in global information modeling, calculation complexity and precision improvement, the present embodiment proposes a kind of high-efficiency optical flow estimation method based on Mamba. The Mamba architecture used has linear calculation complexity, which can greatly reduce the consumption of calculation resources and inference delay while realizing the ability of global information modeling. This method can effectively reduce the inference delay and calculation resource consumption while ensuring high-precision optical flow estimation, and is suitable for visual scenes with high real-time and precision requirements.

[0054] Figure 1 The overall flowchart of the high-efficiency optical flow estimation method based on Mamba of the present embodiment is shown, Figure 2 The model structure schematic diagram of the present application is shown, which specifically includes the following steps:

[0055] Step 101: feature extraction, normalizing and size aligning the adjacent two frames of images, and extracting dense feature maps with fixed down-sampling rate through a convolutional encoder with shared weights;

[0056] Step 102: feature enhancement, sending the two frames of features into a multi-level state space enhancement module, first aggregating long-range context by intra-frame unit, then interacting and fusing by cross-frame unit, introducing spatial neighborhood state fusion mechanism in the state updating process, so that the features can combine sequence dependence and two-dimensional spatial context at the same time, and then completing channel reorganization and residual correction through a multi-layer perceptron to obtain enhanced features;

[0057] Step 103: initial optical flow estimation, constructing a four-dimensional cost volume on a low resolution and applying exponential normalization along the target coordinate dimension to obtain a matching probability distribution, performing weighted averaging on the target coordinate grid according to the distribution to obtain the corresponding coordinates, and then taking the difference between the source coordinates and the corresponding coordinates to obtain the initial optical flow;

[0058] Step 104: autoregressive refinement, sending the initial optical flow and the context and local correlation after spatial weighted fusion guided by attention into an autoregressive refinement module for a small number of fixed iterations, and restoring to the target resolution by convex combination up-sampling, and outputting the final optical flow.

[0059] Step 101 specifically includes:

[0060] (1) The convolutional encoder based on the ResNet34 structure is used to extract features of the input continuous two frames of images respectively. The encoder parameters are shared between the two frames of images, ensuring the consistency of the feature extraction process.

[0061] (2) The convolutional encoder uses pre-trained weights of a public dataset (ImageNet) for parameter initialization, and is further optimized in the end-to-end training process. The encoder is composed of multiple residual modules, which can effectively improve the feature expression ability and model generalization performance.

[0062] (3) After the above-mentioned encoder multi-layer convolution and down-sampling processing, a single-scale high-dimensional dense feature map is output, which is 1 / 8 of the original input image in height and width, which is used to provide basic feature expression for subsequent feature enhancement and optical flow estimation.

[0063] Step 102 specifically comprises:

[0064] (1) Figure 3 The overall flow of the multi-level feature enhancement module of the application is shown, which specifically comprises the following steps: first, the input two frames of high-dimensional features pass through the intra-frame Mamba unit, which further introduces a two-dimensional space scanning mechanism on the basis of traditional state space recursion, realizing global modeling and context information aggregation within the single frame; then, the two frames of features are dynamically interacted and fused through the cross-frame Mamba unit, fully capturing the spatio-temporal dependence relationship and cross-frame motion features; an MLP module is introduced after each cross-frame Mamba unit to improve the non-linear expression ability of the features. The above-mentioned units can be stacked in multiple layers, and finally output two frames of high expression features enhanced by intra-frame and cross-frame, providing rich and discriminative feature basis for subsequent optical flow estimation, realizing efficient and accurate end-to-end pixel-level motion estimation.

[0065] (2) The application introduces a self-supervised feature enhancement mechanism of the Mamba state space model in the feature enhancement stage. Mamba is a kind of state space neural network structure that can efficiently model long-distance dependencies with linear computational complexity. For the high-dimensional feature map output by the encoder for each frame (wherein ), this step first inputs it into the Mamba state space model, and uses a selective scanning strategy to sequentially perform forward and backward state updates in the spatial dimension. The state update unit includes learnable parameters 、 、 、 , and the state recursion equation is as follows:

[0066]

[0067]

[0068] wherein, is the current position input feature, is the hidden state vector, is the output feature.

[0069] On this basis, the application proposes an improved intra-frame Mamba unit, which is different from the traditional recursive method which only depends on the state of the previous moment. In the state updating process, the application further introduces a spatial neighborhood state fusion mechanism. After obtaining the original state variable , the state of the current position and several neighbor states in its two-dimensional spatial neighborhood are fused together to obtain an enhanced structure-aware state , the formula of which is as follows:

[0070]

[0071] wherein, denotes the index of the th neighbor of the position , and is a learnable fusion weight. This aggregation process makes the state not only depend on the sequence information obtained by time recursion, but also integrate the context of its spatial neighborhood.

[0072] Finally, the output feature is given by the enhanced , the formula of which is as follows:

[0073]

[0074] To eliminate the directional bias of one-dimensional spatial scanning and make full use of the context from both sides of the sequence, this step performs forward and reverse state space processing on the same spatial sequence. The forward direction is used to aggregate the cumulative information before the current position, and the reverse direction is used to supplement the clues of the position after it. The two complement each other, forming a symmetric and stable long-range dependence in a non-causal constraint space, and improving consistency and discriminability in boundary and occlusion areas. The bidirectional aggregation relationship is as follows:

[0075]

[0076] The final output is denoted as Through the above process, the intra-frame Mamba unit can fully aggregate the global context information in the feature map space, providing a high-expressive feature basis for cross-frame feature interaction.

[0077] (3) The application further introduces a cross-frame interaction feature enhancement mechanism of the Mamba state space model in the feature enhancement stage, which is used for bidirectional interaction and fusion of two frames of enhanced features under the premise of maintaining linear calculation complexity, so as to depict long-range space-time dependence between frames and improve the stability and discriminability of optical flow estimation. Let the enhanced features of two frames after processing by the intra-frame Mamba unit be and Firstly, taking the first frame as the main branch, the dynamic response more sensitive to cross-frame alignment is extracted through depth separable convolution and nonlinear activation, and the intermediate representation is obtained, and the formula is as follows:

[0078]

[0079] Among them, is the first frame enhanced feature, represents depth separable convolution, is the activation function, is the output feature of the main branch.

[0080] At the same time, taking the second frame as the modulation branch, linear projection is adopted to obtain the cross-frame modulation information that can be injected into the state update, and the formula is as follows:

[0081]

[0082] Among them, is the second frame enhanced feature, is the linear mapping operator, is the output feature of the modulation branch.

[0083] Then, the two branch representations are spliced in the channel dimension and linearly mapped to generate the input-related state space parameter group, and the formula is as follows:

[0084]

[0085] Among them, represents the splicing of the two in the channel dimension, is the linear mapping, , , correspond to the gating, input mapping and readout parameters respectively, which are adaptively changed with the input.

[0086] In the selective scanning process, , , and the input sequence expanded in spatial order (derived from ) are sent into the state space update unit to complete the hidden state update and readout, and the intermediate output is obtained, and the formula is as follows:

[0087]

[0088] wherein, denotes a state space update unit driven based on input related parameters, is an output result. Similarly, the application also introduces spatial neighborhood state aggregation in the state update process of the cross-frame unit, so that the context of adjacent regions can be considered at the same time when modeling the cross-frame dependency, further enhancing the robustness in large displacement and occlusion scenarios.

[0089] Consistent with the intra-frame Mamba unit, to alleviate the directional bias of one-way scanning and obtain temporal symmetry, the application also adopts bidirectional selective scanning in the cross-frame Mamba unit, and the same spatial sequence is processed in the forward and reverse directions. The two-way output is summed and aggregated in the channel dimension after linear output mapping, forming a cross-frame interaction representation.

[0090] (4) The application sets an MLP after the cross-frame interaction feature of the cross-frame Mamba unit, which is used for nonlinear mixing and residual correction of the source feature and the interaction feature in the channel dimension, forming a stable high-expression feature for subsequent global matching and iterative refinement.

[0091] Specifically, the output of the cross-frame Mamba unit is first subjected to linear merging and layer normalization to align the statistical distribution and suppress the drift caused by dynamic interaction; then, after being concatenated with the corresponding source feature in the channel dimension, it is sent to the MLP. The MLP adopts a structure of two layers of linear mapping, nonlinear activation, and layer normalization: first, the number of channels after concatenation is expanded by 4 times and GELU activation is applied, then the number of channels is restored to be consistent with the source feature, and the training and inference process is stabilized through layer normalization. To ensure smooth information path and stable gradient propagation, the output of this unit and the source feature are added in residual to serve as the final enhancement result of this layer.

[0092] Step 103 specifically includes:

[0093] (1) The application constructs a global matching cost volume. Based on the two-frame enhanced features output in step 2, denoted as query feature (from the first frame) and matched feature (from the second frame). For any source pixel position and target pixel position , the scaled dot product similarity is calculated to form a four-dimensional cost volume, and its formula is as follows:

[0094]

[0095] wherein, is the feature channel dimension, denotes the inner product in the channel dimension, and the above formula That is, the similarity mapping set of each source pixel on the whole target image.

[0096] (2) The application probabilizes the cost volume into a matching distribution. The cost volume is normalized on the target coordinate dimension to obtain the matching probability distribution of each source pixel, and the formula is as follows:

[0097]

[0098] Wherein, indicates that the similarity of all under the fixed is normalized, and the probability distribution summed to 1 is output , which is used to describe the matching confidence of the source pixel at each position of the target image.

[0099] (3) The application obtains the corresponding coordinates and generates the initial optical flow according to the matching distribution. The two-dimensional coordinate grid of the target image is weighted and averaged according to the matching distribution to obtain the corresponding coordinates of the source pixel in the target image, and then the initial optical flow field is obtained by subtracting the source coordinates, and the formula is as follows:

[0100]

[0101]

[0102] Wherein, and are two-dimensional coordinate representations of target and source images respectively, is the corresponding coordinate obtained by weighting, is the initial optical flow output by this step, which is used as the input of the subsequent refinement stage.

[0103] Step 104 specifically comprises:

[0104] (1) Figure 4 The iterative flow diagram of the autoregressive refinement module of the application is shown, which specifically comprises the following steps: first, motion features are constructed according to the initial optical flow. Under the guidance of the initial optical flow obtained in step 3, local correlation is calculated in the neighborhood of the second frame feature map, and is jointly coded with the optical flow to obtain motion representation , and the formula is as follows:

[0105]

[0106]

[0107] where, denotes the local correlation calculated with a window of radius is the correlation feature obtained by convolving with is the displacement feature obtained by convolving with denotes the channel-wise concatenation, is the combination of convolution and nonlinear mapping. Thus, the , which can express the local displacement and correlation, is obtained.

[0108] (2) The present application realizes multi-source information fusion by attention-guided aggregation. On the basis of the obtained in (1), the features output by step 2 and the current hidden state are combined to highlight the region related to motion in space through adaptive weighting, forming the aggregated feature for decoding, whose formula is as follows:

[0109]

[0110]

[0111] where, is the combination of shallow convolution and nonlinear mapping, which is used to generate the weight of three inputs at each spatial position, is normalized in the input channel dimension, so that , denotes , and , denotes element-wise multiplication. In this way, the motion, context and historical memory are organically unified to provide stable input for iterative refinement.

[0112] (3) The present application carries out autoregressive refinement based on Mamba. Taking as the time series input, the hidden state is updated by the Mamba unit and the optical flow increment is predicted, which is superimposed on the current optical flow to form a new estimate, whose formula is as follows:

[0113]

[0114]

[0115] ​​​​​​

[0116] where, is two layers Convolutional mapping outputs the optical flow increment, the initial term is taken as . To improve the differential Mamba, the unit performs two independent Mamba mappings in parallel when updating the state:

[0117]

[0118]

[0119] Then after normalization, the differential fusion is performed to obtain the final hidden state:

[0120]

[0121] where, is a learnable weight, denotes a normalization function.

[0122] The design idea of differential Mamba is that through two parallel mappings and differential operation, the hidden state can highlight the significant motion patterns in the input, while suppressing redundant or noise features, avoiding the energy accumulation and state diffusion problems of traditional single-path recursion in long sequence processing. The normalization process further stabilizes the numerical range, ensuring that the state distribution of different positions and different iteration steps remains consistent, thereby improving the training convergence and inference robustness of the model.

[0123] Thus, the aggregation result gradually corrects the initial estimate through the autoregressive mechanism.

[0124] (4) The present application restores the resolution and outputs the result by convex combination upsampling. The refined low-resolution optical flow is upsampled by convex combination through learnable weights, restored to the target resolution and used as the final output, and the formula is as follows:

[0125]

[0126] where, denotes the convex combination upsampling in the neighborhood according to the prediction weight (the weight is normalized in the neighborhood), is the total number of refinement iterations.

[0127] At this point, steps 101 to 104 form a continuous link of "construction-fusion-refinement-recovery", and output the high-precision optical flow result.

[0128] Embodiment 2

[0129] As Figure 5As shown, the embodiment provides a Mamba-based efficient optical flow estimation device, comprising a memory and one or more processors, the memory storing executable code, and the one or more processors executing the executable code to implement the Mamba-based efficient optical flow estimation method of embodiment 1.

[0130] Embodiment 3

[0131] The embodiment relates to a computer-readable storage medium, which stores a program, and the program is executed by a processor to implement the Mamba-based efficient optical flow estimation method of embodiment 1.

[0132] The content described in the embodiments of the present specification is only a list of implementation forms of the inventive concept, and the protection scope of the present application should not be regarded as being limited to the specific forms stated in the embodiments, and the protection scope of the present application also extends to equivalent technical means that can be thought of by those skilled in the art according to the inventive concept.

Claims

1. A method for efficient optical flow estimation based on Mamba, characterized in that, The application relates to a method for estimating optical flow, comprising the following steps: Collecting two adjacent images to be processed and performing normalization and size alignment processing to form an input image pair with unified resolution and dynamic range; Respectively performing feature extraction on the image pair, and outputting a single-scale high-dimensional dense feature map under a fixed down-sampling rate by using a convolutional encoder with shared weights; Inputting the two-frame dense features into a multi-level feature enhancement module, first aggregating long-range context within the frame, then performing interactive fusion in the cross-frame dimension, and introducing a spatial neighborhood state fusion mechanism in the state updating process; Constructing a four-dimensional cost volume at the low-resolution feature level to represent the correspondence relationship from the source pixel to the target pixel, performing softmax normalization on the target coordinate dimension to obtain a matching probability distribution, and performing weighted averaging on the target coordinate grid according to the distribution to obtain the corresponding coordinates, and then performing difference between the corresponding coordinates and the source coordinates to obtain the initial optical flow; Encoding the initial optical flow and the local correlation of the target feature into motion representation, and inputting the motion representation and the context feature into an attention-guided aggregator for spatial adaptive weighted fusion to obtain aggregated features for refinement; Inputting the aggregated features into an autoregressive refinement module based on differential Mamba, and outputting the optical flow increment step by step according to a preset small number of fixed iteration steps and superimposing the optical flow increment on the current estimation to obtain a more accurate optical flow field; Restoring the refined low-resolution optical flow to the target resolution by convex combination up-sampling, and combining local continuity constraints to complete the final result reconstruction and boundary consistency processing; Outputting the final optical flow result.

2. The method of claim 1, wherein, The feature extraction comprises the following steps: using a shallow residual network with shared weights as an encoder to output a single-scale high-dimensional feature under a fixed down-sampling rate, and balancing the representation capability and the computational complexity.

3. The method of claim 1, wherein, The encoder is composed of a stack of basic residual units, and normalization and nonlinear activation are combined to stabilize the training and inference; In order to facilitate transfer learning and multi-channel input, the encoder supports loading general pre-trained weights and adapting the first layer weights; Channel alignment and light fusion are arranged at the tail of the encoder to form a unified dimensional feature representation for subsequent enhancement and matching stages.

4. The method of claim 1, wherein, The multi-level feature enhancement module is composed of a cascade of multi-layer intra-frame Mamba units and cross-frame Mamba units, and a light MLP is introduced after each layer to perform channel reordering and residual correction, so as to enhance the discriminability without significantly increasing the constant term overhead.

5. The method of claim 1, wherein, Global matching is realized in a global correlation manner on the low-resolution feature, the global correlation matrix is calculated after the spatial dimension of the two features is flattened, and the global correlation matrix is scaled by the square root of the channel number, so that a cost volume representing the correspondence relationship between the source pixel and the target pixel is obtained; The cost volume is subjected to softmax normalization in the target coordinate dimension to obtain a matching probability distribution, and the corresponding coordinates are obtained by weighted averaging on the target coordinate grid according to the distribution; The corresponding coordinates and the source coordinates are subtracted to obtain the initial optical flow, and bidirectional optical flow is output by simultaneously calculating the forward and reverse correlations when needed.

6. The method of claim 1, wherein, The autoregressive refinement module adopts a fixed small number of iterations to control the reasoning time delay and the amount of calculation, the iteration step number is set as a predetermined constant during deployment, and is updated by accumulating after each step of outputting the optical flow increment by the differential Mamba-based decoding unit.

7. The method of claim 1, wherein, The upsampling is realized by convex combination of the neighborhood optical flow through the learned local weights, realizing the continuity reconstruction of sub-pixel accuracy.

8. The method of claim 3, wherein, The training adopts an end-to-end joint optimization strategy, simultaneously improving the collaborative effect of the four stages of coding, enhancement, matching and refinement, the data enhancement includes brightness and contrast disturbance, random cropping and pyramid scaling, to enhance the generalization ability to large displacement, weak texture and repetitive texture scenes, and the loss design includes a reprojection consistency term, a robustness term for occlusion and boundary, and a smoothing regularization term.

9. A high efficiency optical flow estimation device based on Mamba, characterized in that, The memory stores executable code, and the one or more processors execute the executable code to implement the Mamba-based efficient optical flow estimation method in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, The program is stored on the memory and executed by the processor to implement the Mamba-based efficient optical flow estimation method in any one of claims 1-8.

Images