Optical flow estimation method and system fusing mamba with visual based model knowledge

Abstract

This invention belongs to the field of computer vision and deep learning technology, specifically involving an optical flow estimation method and system that integrates Mamba and visual foundation model knowledge. The method includes: using a convolutional neural network to extract downsampled features from two adjacent input frames to obtain local texture features; downsampling the first frame to obtain contextual features; simultaneously, using a pre-trained visual model to extract global semantic features from two adjacent frames, and performing adaptive fusion enhancement through an adaptive semantic-texture feature fusion module to obtain semantically enhanced image coding feature pairs; constructing a relevant volume through pixel-by-pixel dot product operations; and finally, based on the obtained relevant volume and contextual features, iteratively optimizing the output optical flow through a loop iterative update module. This invention solves the problems of unstable feature representation in low-texture, repetitive texture, or occluded regions, lack of adaptive modeling ability, difficulty in generalizing to different scenes, and inability to balance model performance and efficiency.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and deep learning technology, and more specifically, relates to an optical flow estimation method and system that integrates Mamba and visual fundamental model knowledge. Background Technology

[0002] Optical flow estimation aims to estimate the motion vector of each pixel in two adjacent frames of an image. It is an important foundation for many vision tasks, such as video understanding, object tracking, autonomous driving, motion analysis, and 3D reconstruction.

[0003] Chinese invention patent CN119832029A discloses an implicit image enhancement and optical flow estimation method based on multimodal collaborative optimization. The method includes first acquiring RGB images under normal lighting conditions and corresponding depth maps from the viewpoints; calculating the corresponding 3D point cloud data based on the depth maps and camera intrinsic parameters; synthesizing low-light image data using the RGB images under normal lighting conditions; decomposing high-frequency and low-frequency features of the low-light image using a high-low frequency feature enhancement network for image enhancement; then extracting image features and contextual features from the low-light image; using a 2D-3D feature fusion network to extract 2D image features and 3D point cloud features and aligning and fusing them to obtain image features and contextual features of the normal RGB image, which are used to supervise the feature extraction process of the high-low frequency feature enhancement network; finally, constructing a 4D relevant volume table based on the image features and contextual features of the low-light image, and using GRU to infer optical flow.

[0004] With the development of deep learning, traditional optical flow estimation methods have been gradually replaced by end-to-end methods based on neural networks, which have become the mainstream approach. Methods such as RAFT achieve high-precision optical flow estimation through feature extraction, correlation calculation, and iterative optimization, achieving good results on standard datasets.

[0005] However, existing optical flow estimation methods based on convolutional neural networks (CNNs) still have shortcomings: their receptive fields are limited, making it difficult to stably model pixel correspondences in low-texture, repetitive texture, or occluded regions, leading to unstable optical flow estimation; at the same time, due to the limited distribution of training data, the models have insufficient generalization ability in complex scenes such as illumination changes, scale changes, and dynamic occlusion. To improve model performance, some studies have introduced self-attention mechanisms or Transformer structures to enhance global feature interactions, but these methods usually have high computational costs and relatively simple feature fusion methods, making it difficult to take into account both local details and global semantics, and still have limitations in terms of inference efficiency and generalization performance.

[0006] In recent years, the self-supervised visual model DINOv3 has been able to extract features with global semantic consistency, while the state-space model Mamba can model long-range dependency information with low computational cost. However, these two models have not yet been efficiently combined in optical flow estimation tasks. Therefore, there is an urgent need for a method that can adaptively fuse global semantic features and local texture features, while maintaining inference efficiency and improving the stability and generalization ability of optical flow estimation. Summary of the Invention

[0007] This invention aims to overcome at least one of the defects of the prior art and proposes an optical flow estimation method and system that integrates Mamba and visual basic model knowledge. It solves the problem of unstable matching of existing optical flow estimation methods in low-texture regions, repetitive texture regions and occluded scenes, and improves the generalization performance in unknown scenes while maintaining inference efficiency.

[0008] The detailed technical solution of this invention is as follows:

[0009] An optical flow estimation method integrating Mamba and visual fundamental model knowledge, the method comprising:

[0010] Step S1: Use a convolutional neural network to perform downsampling feature extraction on two adjacent frames of the input image to obtain the local texture features of the two adjacent frames; and downsample the first frame of the two adjacent frames to obtain the context features.

[0011] Step S2: Use the parameter-freezing pre-trained visual model to extract the global semantic features of each of the two adjacent frames;

[0012] Step S3: Adaptively fuse and enhance the local texture features and global semantic features of two adjacent frames to obtain the semantically enhanced image coding features of the two adjacent frames.

[0013] Step S4: Construct a four-dimensional correlation volume by performing pixel-by-pixel dot product operations on the semantically enhanced image coding features of two adjacent frames.

[0014] Step S5: Finally, based on the obtained four-dimensional related volume and context features, the output optical flow is iteratively optimized through iterative updates.

[0015] Furthermore, step S3 specifically includes:

[0016] Step S31: First, analyze the global semantic features. Perform feature preprocessing to enable global semantic features of adjacent frames. With local texture Feature alignment involves upsampling global semantic features to the same resolution as local texture features using bilinear interpolation, and then aligning the channel dimensions using 1×1 convolution.

[0017] Step S32: Then, the aligned global semantic features are... With local texture features Adaptive semantic texture fusion is performed to obtain information aggregation features. ;

[0018] Next, information aggregation features will be used. Compared with global semantic features after SILU activation and local texture features Multiplication yields information interaction characteristics. and ;

[0019] Then by introducing State space structure, for global semantic features With local texture features Perform long-range dependency modeling and interact with the information features. and Corresponding additions yield adaptive complementary features. and This enables dynamic information transfer across modalities, specifically as follows:

[0020] (1);

[0021] (2);

[0022] In formula (1)-(2), This indicates a pixel-by-pixel addition operation. Indicates based on A complete module;

[0023] Step S33: The adaptive complementary features and The image coding features are fused using a lightweight fusion module consisting of depthwise convolutions and ordinary convolutions to obtain the final semantically enhanced features. Specifically:

[0024] (3);

[0025] (4);

[0026] In formulas (3)-(4), These are the initial fusion features generated in the lightweight fusion module. This indicates a channel splicing operation. This indicates a convolution operation.

[0027] Furthermore, the adaptive semantic texture fusion specifically includes:

[0028] global semantic features With local texture features Perform element-wise addition, then... Layer normalization, followed by depthwise convolution activation and Module processing to obtain information aggregation features The specific formula is as follows:

[0029] (5);

[0030] In formula (5), express Layer normalization operation, Indicates a linear layer. Represents depthwise convolution. This represents the activation function. Indicates 2D selective scan operation. This represents the characteristics of information aggregation.

[0031] Furthermore, the aforementioned The module's processing procedure is as follows:

[0032] Token embedding is performed on the input features to obtain four feature input sequences according to different scanning directions:

[0033] (6);

[0034] In formula (6), Indicates input features, Indicates feature token embedding, The sequences to be scanned represent four different directions;

[0035] After updating the scanned sequences in the four different directions using the S6 scanning module, the updated scanned sequences are obtained. ;

[0036] The updated scan sequence will be obtained. After performing sequence aggregation, information aggregation features are obtained. The obtained information aggregation features It contains both global semantic and local texture information, and has undergone... Enhanced processing can provide a foundation for subsequent feature complementarity fusion.

[0037] Furthermore, the aforementioned The state-space structure includes two branches: one branch is... , , , and Another branch is and Specifically:

[0038] (7);

[0039] In formula (7), express Module input, This indicates the output.

[0040] Furthermore, the step of obtaining global semantic features through the frozen parameter visual model DINOv3 specifically involves:

[0041] The input image is two adjacent frames that have been normalized and then processed. The encoder extracts its own global semantic features. and Specifically:

[0042] (8);

[0043] In formula (8), Indicates the input image. This indicates a normalization operation. This indicates a visual model with pre-trained weights loaded, and its output... Represents global semantic features (D=384), global semantic features include and .

[0044] The output of this step The features contain rich semantic consistency information and have global semantic awareness of object regions, foreground, background and boundaries, thus making up for the feature instability of CNN in low texture and repetitive texture scenes.

[0045] Furthermore, the step of iteratively optimizing the output optical flow based on the obtained four-dimensional related volume and context features through cyclic iteration specifically includes:

[0046] In each iteration, the current predicted optical flow, the matching information extracted from the four-dimensional related volume, and the context features are taken as input. The hidden state is updated by a convolutional gated recurrent unit (GRU), and the optical flow residual is predicted based on the updated hidden state. The optical flow residual is added to the current predicted optical flow to obtain the optical flow prediction result for the next iteration.

[0047] This cyclic update process is repeated a preset number of times to gradually refine the optical flow estimation results, and finally outputs the optical flow F.

[0048] In another aspect of the invention, a system is provided for implementing an optical flow estimation method that integrates Mamba and visual fundamental model knowledge, the system comprising:

[0049] At least one processor; and

[0050] A memory that stores instructions that, when executed by the at least one processor, cause the at least one processor to perform the optical flow estimation method as described above, which integrates knowledge of Mamba and the visual fundamental model.

[0051] In another aspect of the invention, a computer-readable storage medium is also provided, which stores executable instructions that, when executed, cause the machine to perform the optical flow estimation method as described above, which integrates knowledge of Mamba and a visual fundamental model.

[0052] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0053] (1) The optical flow estimation method and system provided by this invention, which integrates Mamba and visual basic model knowledge, introduces global semantic features of the pre-trained visual model DINOv3. That is, by using the pre-trained visual model DINOv3 with frozen parameters, high-level global semantic features are extracted from the image, which complement local texture features. Through this design, this invention can maintain stable feature matching in complex environments such as illumination changes, scale changes, and partial occlusion, effectively improving the robustness and cross-scene generalization ability of optical flow estimation.

[0054] (2) The optical flow estimation method and system that integrates Mamba and visual basic model knowledge provided by the present invention designs an adaptive feature fusion module based on the Mamba architecture. In view of the problem of feature distribution differences and redundant information between global semantic features and original local texture features, an adaptive semantic texture feature fusion module based on Mamba is proposed. Feature aggregation is achieved through lightweight deep convolution and state space unit (SS2D). Without significantly increasing the computational load of the model, effective complementarity and adaptive fusion of dual-path features are achieved. Compared with traditional Transformer or attention fusion modules, while maintaining low memory usage and computational overhead, the global consistency and detail sensitivity of feature expression are significantly improved.

[0055] (3) The optical flow estimation method and system provided by this invention, which integrates Mamba and visual basic model knowledge, introduces global semantic features of the pre-trained visual model DINOv3, an adaptive feature fusion module based on the Mamba architecture, and utilizes the "knowledge reserve" of the pre-trained DINOv3 to provide semantic priors for the optical flow estimation model. Combined with the streaming feature modeling characteristics of Mamba, the overall framework is superior to the scheme that introduces a large Transformer module in terms of inference speed and memory usage, thus achieving a dual balance between performance and efficiency, and has good deployment value. It can also maintain matching stability in low-texture, repetitive texture, and occluded scenes, significantly improve the accuracy and generalization ability of optical flow estimation in unknown scenes, and maintain high inference efficiency and lightweight feature fusion capability, which has significant technological progress and application promotion value. Attached Figure Description

[0056] Figure 1 This is a flowchart of the optical flow estimation method described in this invention.

[0057] Figure 2 It is based on Embodiment 1 of the present invention A schematic diagram of the adaptive semantic texture feature fusion module.

[0058] Figure 3 This is a schematic diagram of the SS2D module structure in Embodiment 1 of the present invention. Detailed Implementation

[0059] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0060] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0061] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0062] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0063] Example 1

[0064] To address the problems of unstable matching in low-texture, repetitive texture, and occlusion regions, as well as insufficient generalization in complex scenarios, in existing optical flow estimation methods, the present invention proposes an optical flow estimation method and system that integrates Mamba and visual foundation model knowledge. This method innovatively introduces the global semantic features extracted by the pre-trained vision model DINOv3 into the optical flow estimation framework and designs an adaptive feature fusion module based on the Mamba structure to achieve efficient dynamic fusion between local texture features and global semantic features while maintaining linear complexity and reducing computational overhead. Through the above design, without significantly increasing the computational overhead, the present invention effectively improves the robustness of the model to weak texture, illumination changes, and dynamic occlusion scenarios, enhancing the stability and generalization ability of optical flow estimation. At the same time, the fusion structure based on the Mamba state space model can capture long-range dependence information while maintaining lightweight, further balancing model performance and inference efficiency.

[0065] Reference Figure 1 In this embodiment, an optical flow estimation method that integrates Mamba and visual foundation model knowledge is provided. The method includes:

[0066] Step S1: Perform downsampling feature extraction on two adjacent input images and through a convolutional neural network to obtain local texture features and ; and downsample the first image to obtain context features .

[0067] Specifically, perform multi-layer downsampling and convolutional operation feature extraction on two adjacent input images through a residual convolutional network ( and ) to obtain the local texture features and of the two adjacent images and the context feature ; the context feature is obtained by downsampling , and the specific operation is as follows:

[0068] (9);

[0069] (10);

[0070] In formulas (9)-(10), represents the input image, represents the local texture feature extraction module, and represent local texture features (D = 256), where D represents the number of channels, For the context feature extraction module, and The structures are the same but the weights are independent. Indicates contextual features.

[0071] Step S2: Extract images from two adjacent frames using a pre-trained visual model with parameter freezing. and global semantic features and ;

[0072] Preferably, a pre-trained large visual model is loaded first. The model parameters remain frozen and are used only as a semantic feature extractor.

[0073] Specifically, the acquisition of global semantic features through a self-supervised visual encoder (i.e., the frozen parameter visual model DINOv3) involves the following steps:

[0074] The input image is two adjacent frames that have been normalized and then processed. The encoder extracts global semantic features from two adjacent frames. and Specifically:

[0075] (8);

[0076] In formula (8), Indicates the input image. This indicates a normalization operation. This indicates a visual model with pre-trained weights loaded, and its output... Representing global semantic features (D=384), in this embodiment, global semantic features include and .

[0077] The output of this step The features contain rich semantic consistency information and have global semantic awareness of object regions, foreground, background and boundaries, thus making up for the feature instability of CNN in low texture and repetitive texture scenes.

[0078] Step S3: Adaptively enhance and fuse the local texture features with the global semantic features to obtain semantically enhanced image coding features. and This module can achieve efficient feature complementarity fusion while maintaining inference efficiency.

[0079] Specifically, through based on The adaptive semantic texture feature fusion module adaptively enhances and fuses the local texture features with the global semantic features, such as... Figure 2As shown, based on The adaptive semantic texture feature fusion module consists of four parts: feature input, feature complementarity enhancement module, lightweight fusion module, and feature output.

[0080] Step S3 specifically includes:

[0081] Step S31: Feature input;

[0082] First, global semantic features Perform feature preprocessing to enable global semantic features of adjacent frames. With local texture Feature alignment, i.e., image of and Alignment, Image of and Alignment, Local Texture Including local textures of two adjacent frames and ;

[0083] Preferably, the global semantic features are upsampled to the same resolution as the local texture features using bilinear interpolation, and then 1×1 convolution is used for channel dimension alignment. The specific alignment operation is as follows:

[0084] (11);

[0085] at this time, and Alignment This represents the upsampling function using bilinear interpolation. This indicates a convolutional module with a kernel size of 1×1.

[0086] Step S32: Processing of the feature complementarity enhancement module;

[0087] Then, the global semantic features of each of the two adjacent aligned frames are... With local texture features Adaptive semantic texture fusion is performed to obtain information aggregation features between two adjacent frames. In this embodiment include and ;

[0088] Next, the information from two adjacent frames is aggregated as features. Compared with global semantic features after SILU activation and local texture features Multiplication yields the information interaction features of two adjacent frames. and The specific operation is as follows:

[0089] (12);

[0090] (13);

[0091] In formula (12)-(13), This represents pixel-by-pixel dot product operation. and It indicates the characteristics of information interaction.

[0092] By introducing State space structure, for global semantic features With local texture features Perform long-range dependency modeling and interact with the information features. and Corresponding additions yield adaptive complementary features. and This enables dynamic information transfer across modalities. The specific operation is as follows:

[0093] (1);

[0094] (2);

[0095] In formula (1)-(2), This indicates a pixel-by-pixel addition operation. Indicates based on A complete module.

[0096] State space structure as follows Figure 2 As shown, it includes two branches: one branch is , , , and Another branch is and The specific operation is as follows:

[0097] (7);

[0098] in, express The input of the module is , , This indicates the output.

[0099] Specifically, the adaptive semantic texture fusion is as follows:

[0100] First, global semantic features With local texture features Perform element-wise addition, then... Layer normalization, followed by depthwise convolution activation and Module processing to obtain information aggregation features The specific formula is as follows:

[0101] (5);

[0102] in, express Layer normalization operation, Indicates a linear layer. Represents depthwise convolution. This represents the activation function. Indicates 2D selective scan operation. This represents the characteristics of information aggregation.

[0103] Preferably, the See module Figure 3 Specifically:

[0104] First, the input feature tokens are embedded, resulting in four feature input sequences according to different scanning directions:

[0105] (6);

[0106] In formula (6), Indicates input features, Indicates feature token embedding, The sequences to be scanned represent four different directions.

[0107] After updating the scanned sequences in the four different directions using the S6 scanning module, the updated scanned sequences are obtained. S6 is an industry term, a key term in Mamba technology. The specific operation of the S6 scanning module is as follows:

[0108] (14);

[0109] (15);

[0110] (16);

[0111] (17);

[0112] (18);

[0113] (19);

[0114] In formulas (14)-(19), where, represent Input features during operation Its sequence length is ; Represents the time scale parameter; and These represent the linear projections of the input features, respectively. , , Represents three different parameter matrices; Represents the state matrix. and They are respectively and Discrete parameters; Represents a jump link, for The intermediate state at a given moment. represent Output characteristics of the operation.

[0115] The above obtained updated scan sequence After performing sequence aggregation, information aggregation features are obtained. The obtained information aggregation features It contains both global semantic and local texture information, and has undergone... Enhanced processing can provide a foundation for subsequent feature complementarity fusion.

[0116] Step S33: Implementation of the lightweight fusion module;

[0117] The adaptive complementary features and The image coding features are fused using a lightweight fusion module consisting of depthwise convolutions and ordinary convolutions to obtain the final semantically enhanced features. Specifically:

[0118] (3);

[0119] (4);

[0120] In formulas (3)-(4), These are the initial fusion features generated in the lightweight fusion module. This indicates a channel splicing operation. This indicates a convolution operation.

[0121] Step S4: Based on the semantically enhanced image coding feature pair described in step S3 and A four-dimensional related volume is constructed through pixel-by-pixel dot product operations. Figure 1 middle This represents pixel-by-pixel dot product operation;

[0122] The specific operation is represented as follows:

[0123] (20);

[0124] In formula (20), Represents the four-dimensional related volume, where each location Represents the first frame pixels With the second frame pixels Feature similarity between This represents the vector dot product operation. Indicates the position in the features of the first frame. This indicates the position within the features of the second frame.

[0125] Step S5: Finally, based on the obtained four-dimensional related volume and context features, the output optical flow is iteratively updated and optimized through loop iteration. .

[0126] Specifically, the four-dimensional related volume Contextual features and initial optical flow The input loop update module is used to iteratively update the predicted optical flow, and finally the updated predicted optical flow is upsampled to obtain the original resolution optical flow map.

[0127] Preferably, the cyclic update module is a GRU-based cyclic update module, which is a commonly used iterative optimization structure in existing optical flow estimation methods. It performs multiple iterative updates on the current optical flow while maintaining the spatial resolution.

[0128] In each iteration, the current predicted optical flow (initial optical flow in the first iteration), the matching information extracted from the four-dimensional related volume, and the context features are used as inputs. The hidden state is updated by a convolutional gated recurrent unit (GRU), and the optical flow residual is predicted based on the updated hidden state. The optical flow residual is added to the current predicted optical flow to obtain the optical flow prediction result for the next iteration.

[0129] This cyclic update process is repeated a preset number of times to gradually refine the optical flow estimation results, and finally outputs the optical flow F.

[0130] In summary, this invention innovatively introduces global semantic features from the pre-trained visual model DINOv3 into the optical flow estimation framework and combines it with an adaptive feature fusion module based on the Mamba structure to achieve efficient fusion of local texture features and global semantic features. This approach not only improves the model's matching stability in low-texture, repetitive texture, and occluded regions but also enhances the generalization performance of optical flow estimation in complex and unknown scenarios. Compared with existing methods, this invention achieves better accuracy and robustness while maintaining inference efficiency, demonstrating good engineering applicability and promotional value.

[0131] Example 2

[0132] This embodiment provides a system for implementing an optical flow estimation method that integrates knowledge from Mamba and visual fundamental models. The apparatus includes:

[0133] At least one processor; and

[0134] A memory that stores instructions that, when executed by the at least one processor, cause the at least one processor to perform the optical flow estimation method as described above, which integrates knowledge of Mamba and the visual fundamental model.

[0135] In this embodiment, electronic devices include, but are not limited to: personal computers, server computers, workstations, desktop computers, laptop computers, notebook computers, mobile computing devices, smartphones, tablet computers, cellular phones, personal digital assistants (PDAs), handheld devices, messaging devices, wearable computing devices, consumer electronic devices, etc.

[0136] Example 3

[0137] This embodiment also provides a computer-readable storage medium storing executable instructions that, when executed, cause the machine to perform the optical flow estimation method described above, which integrates knowledge from Mamba and the visual fundamental model.

[0138] Specifically, a system or apparatus equipped with a readable storage medium may be provided, on which software program code implementing the functions of any of the embodiments described above is stored, and the computer or processor of the system or apparatus can read and execute the instructions stored in the readable storage medium.

[0139] In this case, the program code itself, which can be read from the readable medium, can perform the functions of any of the above embodiments, and therefore the computer-readable code and the readable storage medium storing the computer-readable code constitute a part of this specification.

[0140] Examples of readable storage media include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD-RW), magnetic tapes, non-volatile memory cards, and ROMs. Alternatively, program code can be downloaded from a server computer or the cloud via a communication network.

[0141] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0142] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0143] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0144] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0145] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the technical solutions of the present invention, and are not intended to limit the specific implementation of the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the claims of the present invention should be included within the protection scope of the claims of the present invention.

Claims

1. An optical flow estimation method integrating Mamba and visual fundamental model knowledge, characterized in that, The method includes: Step S1: Use a convolutional neural network to perform downsampling feature extraction on two adjacent frames of the input image to obtain the local texture features of the two adjacent frames; and downsample the first frame of the two adjacent frames to obtain the context features. Step S2: Use the parameter-freezing pre-trained visual model to extract the global semantic features of each of the two adjacent frames; Step S3: Adaptively fuse and enhance the local texture features and global semantic features of two adjacent frames to obtain the semantically enhanced image coding features of the two adjacent frames. Step S3 specifically includes: Step S31: First, analyze the global semantic features. Perform feature preprocessing to enable global semantic features of adjacent frames. With local texture Feature alignment; Step S32: Then, the aligned global semantic features are... With local texture features Adaptive semantic texture fusion is performed to obtain information aggregation features. ; Next, information aggregation features will be used. Compared with global semantic features after SILU activation and local texture features Multiplication yields information interaction characteristics. and ; By introducing State space structure, for global semantic features With local texture features Perform long-range dependency modeling and interact with the information features. and Corresponding additions yield adaptive complementary features. and This enables dynamic information transfer across modalities, specifically as follows: （1）； （2）； In formula (1)-(2), This indicates a pixel-by-pixel addition operation. Indicates based on A complete model; Step S33: The adaptive complementary features and A lightweight fusion module consisting of depthwise convolutions and ordinary convolutions is used to fuse the images, obtaining the final semantically enhanced image coding features for each of two adjacent frames. Specifically: （3）； （4）； In formulas (3)-(4), These are the initial fusion features generated in the lightweight fusion module. This indicates a channel splicing operation. Indicates the convolution operation; The adaptive semantic texture fusion specifically refers to: The global semantic features of two adjacent frames With local texture features Perform element-by-element addition, then... Layer normalization, followed by depthwise convolution activation and Module processing to obtain information aggregation features The specific formula is as follows: （5）； In formula (5), express Layer normalization operation, Indicates 2D selective scan operation. Represents depthwise convolution. This represents the activation function. Indicates a linear layer. Represents information aggregation characteristics; Step S4: Construct a four-dimensional correlation volume by performing pixel-by-pixel dot product operations on the semantically enhanced image coding features of two adjacent frames. Step S5: Finally, based on the obtained four-dimensional related volume and context features, the output optical flow is iteratively optimized through iterative updates.

2. The optical flow estimation method fusing Mamba and visual fundamental model knowledge according to claim 1, characterized in that, The The module's processing procedure is as follows: Token embedding is performed on the input features to obtain four feature input sequences according to different scanning directions: （6）； In formula (6), Indicates input features, Indicates feature token embedding, The sequences to be scanned represent four different directions; After updating the scan sequences in the four different directions using the S6 scanning module, the updated scan sequences are obtained. ; The updated scan sequence will be obtained. After performing sequence aggregation, information aggregation features are obtained. The obtained information aggregation features It contains both global semantic and local texture information, and has undergone... Enhanced processing can provide a foundation for subsequent feature complementarity fusion.

3. The optical flow estimation method fusing Mamba and visual fundamental model knowledge according to claim 2, characterized in that, The process of iteratively optimizing the output optical flow based on the obtained four-dimensional related volume and context features through cyclic iteration is as follows: In each iteration, the current predicted optical flow, the matching information extracted from the four-dimensional related volume, and the context features are taken as input. The hidden state is updated by a convolutional gated recurrent unit (GRU), and the optical flow residual is predicted based on the updated hidden state. The optical flow residual is added to the current predicted optical flow to obtain the optical flow prediction result for the next iteration. This cyclic update process is repeated a preset number of times to gradually refine the optical flow estimation results, and finally outputs the optical flow F.

4. The optical flow estimation method fusing Mamba and visual fundamental model knowledge according to claim 1, characterized in that, The The state-space structure includes two branches: one branch is... , , , and Another branch is and Specifically: （7）； In formula (7), express Module input, This indicates the output.

5. The optical flow estimation method fusing Mamba and visual fundamental model knowledge according to claim 1, characterized in that, The process of obtaining global semantic features through the frozen parameter visual model DINOv3 specifically involves: The input image is two adjacent frames that have been normalized and then processed. The encoder extracts its own global semantic features. and Specifically: （8）； In formula (8), Indicates the input image. This indicates a normalization operation. This indicates a visual model with pre-trained weights loaded, and its output... Represents global semantic features. include and .

6. The optical flow estimation method fusing Mamba and visual fundamental model knowledge according to claim 1, characterized in that, Make the global semantic features of two adjacent frames respectively With local texture Feature alignment, specifically: Global semantic features are obtained through bilinear interpolation. Upsampling to local texture features At the same resolution, channel dimensions are aligned using 1×1 convolution.

7. A system for implementing an optical flow estimation method that integrates knowledge from Mamba and visual fundamental models, characterized in that, The system includes: processor; A memory on which computer programs that can run on the processor are stored; When the computer program is executed by the processor, it implements the steps of the optical flow estimation method that integrates Mamba and visual fundamental model knowledge as described in any one of claims 1 to 6.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 6.

Images