Optical flow estimation method and system fusing Mama and visual basis model knowledge

Abstract

The invention belongs to the technical field of computer vision and deep learning, and particularly relates to an optical flow estimation method and system fusing Mama and visual basic model knowledge. The method comprises the following steps: performing down-sampling feature extraction on two adjacent frames of input images by using a convolutional neural network to obtain local texture features; performing down-sampling on the first frame image to obtain context features; meanwhile, extracting global semantic features of two adjacent frames of images by using a pre-trained visual model, and performing adaptive fusion enhancement through an adaptive semantic texture feature fusion module to obtain an image coding feature pair after semantic enhancement; constructing a related volume through pixel-by-pixel dot product operation; and finally, based on the obtained related volume and context features, iteratively optimizing the output optical flow through a loop iteration updating module. The method solves the problems that in a low-texture, repeated-texture or sheltered area, feature expression is unstable, self-adaptive modeling capacity is lacked, different scenes are difficult to generalize, and model performance and efficiency cannot be balanced.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and deep learning technology, and more specifically, relates to an optical flow estimation method and system that integrates Mamba and visual fundamental model knowledge. Background Technology

[0002] Optical flow estimation aims to estimate the motion vector of each pixel in two adjacent frames of an image. It is an important foundation for many vision tasks, such as video understanding, object tracking, autonomous driving, motion analysis, and 3D reconstruction.

[0003] Chinese invention patent CN119832029A discloses an implicit image enhancement and optical flow estimation method based on multimodal collaborative optimization. The method includes first acquiring RGB images under normal lighting conditions and corresponding depth maps from the viewpoints; calculating the corresponding 3D point cloud data based on the depth maps and camera intrinsic parameters; synthesizing low-light image data using the RGB images under normal lighting conditions; decomposing high-frequency and low-frequency features of the low-light image using a high-low frequency feature enhancement network for image enhancement; then extracting image features and contextual features from the low-light image; using a 2D-3D feature fusion network to extract 2D image features and 3D point cloud features and aligning and fusing them to obtain image features and contextual features of the normal RGB image, which are used to supervise the feature extraction process of the high-low frequency feature enhancement network; finally, constructing a 4D relevant volume table based on the image features and contextual features of the low-light image, and using GRU to infer optical flow.

[0004] With the development of deep learning, traditional optical flow estimation methods have been gradually replaced by end-to-end methods based on neural networks, which have become the mainstream approach. Methods such as RAFT achieve high-precision optical flow estimation through feature extraction, correlation calculation, and iterative optimization, achieving good results on standard datasets.

[0005] However, existing optical flow estimation methods based on convolutional neural networks (CNNs) still have shortcomings: their receptive fields are limited, making it difficult to stably model pixel correspondences in low-texture, repetitive texture, or occluded regions, leading to unstable optical flow estimation; at the same time, due to the limited distribution of training data, the models have insufficient generalization ability in complex scenes such as illumination changes, scale changes, and dynamic occlusion. To improve model performance, some studies have introduced self-attention mechanisms or Transformer structures to enhance global feature interactions, but these methods usually have high computational costs and relatively simple feature fusion methods, making it difficult to take into account both local details and global semantics, and still have limitations in terms of inference efficiency and generalization performance.

[0006] In recent years, the self-supervised visual model DINOv3 has been able to extract features with global semantic consistency, while the state-space model Mamba can model long-range dependency information with low computational cost. However, these two models have not yet been efficiently combined in optical flow estimation tasks. Therefore, there is an urgent need for a method that can adaptively fuse global semantic features and local texture features, while maintaining inference efficiency and improving the stability and generalization ability of optical flow estimation. Summary of the Invention

[0007] This invention aims to overcome at least one of the defects of the prior art and proposes an optical flow estimation method and system that integrates Mamba and visual basic model knowledge. It solves the problem of unstable matching of existing optical flow estimation methods in low-texture regions, repetitive texture regions and occluded scenes, and improves the generalization performance in unknown scenes while maintaining inference efficiency.

[0008] The detailed technical solution of this invention is as follows: An optical flow estimation method integrating Mamba and visual fundamental model knowledge, the method comprising: Step S1: Use a convolutional neural network to perform downsampling feature extraction on two adjacent frames of the input image to obtain the local texture features of the two adjacent frames; and downsample the first frame of the two adjacent frames to obtain the context features. Step S2: Use the parameter-freezing pre-trained visual model to extract the global semantic features of each of the two adjacent frames; Step S3: Adaptively fuse and enhance the local texture features and global semantic features of two adjacent frames to obtain the semantically enhanced image coding features of the two adjacent frames. Step S4: Construct a four-dimensional correlation volume by performing pixel-by-pixel dot product operations on the semantically enhanced image coding features of two adjacent frames. Step S5: Finally, based on the obtained four-dimensional related volume and context features, the output optical flow is iteratively optimized through iterative updates.

[0009] Furthermore, step S3 specifically includes: Step S31: First, analyze the global semantic features. Perform feature preprocessing to enable global semantic features of adjacent frames. With local texture Feature alignment involves upsampling global semantic features to the same resolution as local texture features using bilinear interpolation, and then aligning the channel dimensions using 1×1 convolution. Step S32: Then, the aligned global semantic features are... With local texture features Adaptive semantic texture fusion is performed to obtain information aggregation features. ; Next, information aggregation features will be used. Compared with global semantic features after SILU activation and local texture features Multiplication yields information interaction characteristics. and ; Then by introducing State space structure, for global semantic features With local texture features Perform long-range dependency modeling and interact with the information features. and Corresponding additions yield adaptive complementary features. and This enables dynamic information transfer across modalities, specifically as follows: (1); (2); In formula (1)-(2), This indicates a pixel-by-pixel addition operation. Indicates based on A complete module; Step S33: The adaptive complementary features and The image coding features are fused using a lightweight fusion module consisting of depthwise convolutions and ordinary convolutions to obtain the final semantically enhanced features. Specifically: (3); (4); In formulas (3)-(4), These are the initial fusion features generated in the lightweight fusion module. This indicates a channel splicing operation. This indicates a convolution operation.

[0010] Furthermore, the adaptive semantic texture fusion specifically includes: global semantic features With local texture features Perform element-wise addition, then... Layer normalization, followed by depthwise convolution activation and Module processing to obtain information aggregation features The specific formula is as follows: (5); In formula (5), express Layer normalization operation, Indicates a linear layer. Represents depthwise convolution. This represents the activation function. Indicates 2D selective scan operation. This represents the characteristics of information aggregation.

[0011] Furthermore, the aforementioned The module's processing procedure is as follows: Token embedding is performed on the input features to obtain four feature input sequences according to different scanning directions: (6); In formula (6), Indicates input features, Indicates feature token embedding, The sequences to be scanned represent four different directions; After updating the scanned sequences in the four different directions using the S6 scanning module, the updated scanned sequences are obtained. ; The updated scan sequence will be obtained. After performing sequence aggregation, information aggregation features are obtained. The obtained information aggregation features It contains both global semantic and local texture information, and has undergone... Enhanced processing can provide a foundation for subsequent feature complementarity fusion.

[0012] Furthermore, the aforementioned The state-space structure includes two branches: one branch is... , , , and Another branch is and Specifically: (7); In formula (7), express Module input, This indicates the output.

[0013] Furthermore, the step of obtaining global semantic features through the frozen parameter visual model DINOv3 specifically involves: The input image is two adjacent frames that have been normalized and then processed. The encoder extracts its own global semantic features. and Specifically: (8); In formula (8), Indicates the input image. This indicates a normalization operation. This indicates a visual model with pre-trained weights loaded, and its output... Represents global semantic features (D=384), global semantic features include and .

[0014] The output of this step The features contain rich semantic consistency information and have global semantic awareness of object regions, foreground, background and boundaries, thus making up for the feature instability of CNN in low texture and repetitive texture scenes.

[0015] Furthermore, the step of iteratively optimizing the output optical flow based on the obtained four-dimensional related volume and context features through cyclic iteration specifically includes: In each iteration, the current predicted optical flow, the matching information extracted from the four-dimensional related volume, and the context features are taken as input. The hidden state is updated by a convolutional gated recurrent unit (GRU), and the optical flow residual is predicted based on the updated hidden state. The optical flow residual is added to the current predicted optical flow to obtain the optical flow prediction result for the next iteration. This cyclic update process is repeated a preset number of times to gradually refine the optical flow estimation results, and finally outputs the optical flow F.

[0016] In another aspect of the invention, a system is provided for implementing an optical flow estimation method that integrates Mamba and visual fundamental model knowledge, the system comprising: At least one processor; and A memory that stores instructions that, when executed by the at least one processor, cause the at least one processor to perform the optical flow estimation method as described above, which integrates knowledge of Mamba and the visual fundamental model.

[0017] In another aspect of the invention, a computer-readable storage medium is also provided, which stores executable instructions that, when executed, cause the machine to perform the optical flow estimation method as described above, which integrates knowledge of Mamba and a visual fundamental model.

[0018] Compared with the prior art, the beneficial effects of the present invention are as follows: (1) The optical flow estimation method and system provided by this invention, which integrates Mamba and visual basic model knowledge, introduces global semantic features of the pre-trained visual model DINOv3. That is, by using the pre-trained visual model DINOv3 with frozen parameters, high-level global semantic features are extracted from the image, which complement local texture features. Through this design, this invention can maintain stable feature matching in complex environments such as illumination changes, scale changes, and partial occlusion, effectively improving the robustness and cross-scene generalization ability of optical flow estimation.

[0019] (2) The optical flow estimation method and system that integrates Mamba and visual basic model knowledge provided by the present invention designs an adaptive feature fusion module based on the Mamba architecture. In view of the problem of feature distribution differences and redundant information between global semantic features and original local texture features, an adaptive semantic texture feature fusion module based on Mamba is proposed. Feature aggregation is achieved through lightweight deep convolution and state space unit (SS2D). Without significantly increasing the computational load of the model, effective complementarity and adaptive fusion of dual-path features are achieved. Compared with traditional Transformer or attention fusion modules, while maintaining low memory usage and computational overhead, the global consistency and detail sensitivity of feature expression are significantly improved.

[0020] (3) The optical flow estimation method and system provided by this invention, which integrates Mamba and visual basic model knowledge, introduces global semantic features of the pre-trained visual model DINOv3, an adaptive feature fusion module based on the Mamba architecture, and utilizes the "knowledge reserve" of the pre-trained DINOv3 to provide semantic priors for the optical flow estimation model. Combined with the streaming feature modeling characteristics of Mamba, the overall framework is superior to the scheme that introduces a large Transformer module in terms of inference speed and memory usage, thus achieving a dual balance between performance and efficiency, and has good deployment value. It can also maintain matching stability in low-texture, repetitive texture, and occluded scenes, significantly improve the accuracy and generalization ability of optical flow estimation in unknown scenes, and maintain high inference efficiency and lightweight feature fusion capability, which has significant technological progress and application promotion value. Attached Figure Description

[0021] Figure 1 This is a flowchart of the optical flow estimation method described in this invention.

[0022] Figure 2 It is based on Embodiment 1 of the present invention A schematic diagram of the adaptive semantic texture feature fusion module.

[0023] Figure 3 This is a schematic diagram of the SS2D module structure in Embodiment 1 of the present invention. Detailed Implementation

[0024] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0025] It should be noted that the following detailed description is exemplary and is intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the technical field to which the present invention belongs.

[0026] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0027] In the case of no conflict, the embodiments in the present invention and the features in the embodiments can be combined with each other.

[0028] Embodiment 1 In order to solve the problems of unstable matching in low-texture, repetitive texture, and occlusion regions and insufficient generalization in complex scenarios of existing optical flow estimation methods, the present invention proposes an optical flow estimation method and system that combines Mamba and visual foundation model knowledge. This method innovatively introduces the global semantic features extracted by the pre-trained visual model DINOv3 into the optical flow estimation framework and designs an adaptive feature fusion module based on the Mamba structure to achieve efficient dynamic fusion between local texture features and global semantic features while maintaining linear complexity and reducing computational overhead. Through the above design, the present invention effectively improves the robustness of the model to weak texture, illumination changes, and dynamic occlusion scenarios without significantly increasing the computational overhead, enhancing the stability and generalization ability of optical flow estimation. At the same time, the fusion structure based on the Mamba state space model can capture long-range dependence information while maintaining light weight, further balancing model performance and inference efficiency.

[0029] Refer Figure 1 , this embodiment provides an optical flow estimation method that combines Mamba and visual foundation model knowledge, and the method includes: Step S1: Perform downsampling feature extraction on two adjacent input images and through a convolutional neural network to obtain local texture features and ; and downsample the first image to obtain context features .

[0030] Specifically, through residual convolutional networks ( For two adjacent frames of the input image and Multi-layer downsampling and convolution operations are performed to extract features, resulting in local texture features between two adjacent frames. and With context features ; Contextual features are The results are obtained through downsampling, and the specific steps are as follows: (9); (10); In formulas (9)-(10), Indicates the input image. This represents the local texture feature extraction module. and This represents local texture features (D=256), where D represents the number of channels. For the context feature extraction module, and The structures are the same but the weights are independent. Indicates contextual features.

[0031] Step S2: Extract images from two adjacent frames using a pre-trained visual model with parameter freezing. and global semantic features and ; Preferably, a pre-trained large visual model is loaded first. The model parameters remain frozen and are used only as a semantic feature extractor.

[0032] Specifically, the acquisition of global semantic features through a self-supervised visual encoder (i.e., the frozen parameter visual model DINOv3) involves the following steps: The input image is two adjacent frames that have been normalized and then processed. The encoder extracts global semantic features from two adjacent frames. and Specifically: (8); In formula (8), Indicates the input image. This indicates a normalization operation. This indicates a visual model with pre-trained weights loaded, and its output... Representing global semantic features (D=384), in this embodiment, global semantic features include and .

[0033] The output of this step The features contain rich semantic consistency information and have global semantic awareness of object regions, foreground, background and boundaries, thus making up for the feature instability of CNN in low texture and repetitive texture scenes.

[0034] Step S3: Adaptively enhance and fuse the local texture features with the global semantic features to obtain semantically enhanced image coding features. and This module can achieve efficient feature complementarity fusion while maintaining inference efficiency.

[0035] Specifically, through based on The adaptive semantic texture feature fusion module adaptively enhances and fuses the local texture features with the global semantic features, such as... Figure 2 As shown, based on The adaptive semantic texture feature fusion module consists of four parts: feature input, feature complementarity enhancement module, lightweight fusion module, and feature output.

[0036] Step S3 specifically includes: Step S31: Feature input; First, global semantic features Perform feature preprocessing to enable global semantic features of adjacent frames. With local texture Feature alignment, i.e., image of and Alignment, Image of and Alignment, Local Texture Including local textures of two adjacent frames and ; Preferably, the global semantic features are upsampled to the same resolution as the local texture features using bilinear interpolation, and then 1×1 convolution is used for channel dimension alignment. The specific alignment operation is as follows: (11); at this time, and Alignment This represents the upsampling function using bilinear interpolation. This indicates a convolutional module with a kernel size of 1×1.

[0037] Step S32: Processing of the feature complementarity enhancement module; Then, the global semantic features of each of the two adjacent aligned frames are... With local texture features Adaptive semantic texture fusion is performed to obtain information aggregation features between two adjacent frames. In this embodiment include and ; Next, the information from two adjacent frames is aggregated as features. Compared with global semantic features after SILU activation and local texture features Multiplication yields the information interaction features of two adjacent frames. and The specific operation is as follows: (12); (13); In formula (12)-(13), This represents pixel-by-pixel dot product operation. and It indicates the characteristics of information interaction.

[0038] By introducing State space structure, for global semantic features With local texture features Perform long-range dependency modeling and interact with the information features. and Corresponding additions yield adaptive complementary features. and This enables dynamic information transfer across modalities. The specific operation is as follows: (1); (2); In formula (1)-(2), This indicates a pixel-by-pixel addition operation. Indicates based on A complete module.

[0039] State space structure as follows Figure 2 As shown, it includes two branches: one branch is... , , , and Another branch is and The specific operation is as follows: (7); in, express The input of the module is , , This indicates the output.

[0040] Specifically, the adaptive semantic texture fusion is as follows: First, global semantic features With local texture features Perform element-wise addition, then... Layer normalization, followed by depthwise convolution activation and Module processing to obtain information aggregation features The specific formula is as follows: (5); in, express Layer normalization operation, Indicates a linear layer. Represents depthwise convolution. This represents the activation function. Indicates 2D selective scan operation. This represents the characteristics of information aggregation.

[0041] Preferably, the See module Figure 3 Specifically: First, the input feature tokens are embedded, resulting in four feature input sequences according to different scanning directions: (6); In formula (6), Indicates input features, Indicates feature token embedding, The sequences to be scanned represent four different directions.

[0042] After updating the scanned sequences in the four different directions using the S6 scanning module, the updated scanned sequences are obtained. S6 is an industry term, a key term in Mamba technology. The specific operation of the S6 scanning module is as follows: (14); (15); (16); (17); (18); (19); In formulas (14)-(19), where, represent Input features during operation Its sequence length is ; Represents the time scale parameter; and These represent the linear projections of the input features, respectively. , , Represents three different parameter matrices; Represents the state matrix. and They are respectively and Discrete parameters; Represents a jump link, for The intermediate state at a given moment. represent Output characteristics of the operation.

[0043] The above obtained updated scan sequence After performing sequence aggregation, information aggregation features are obtained. The obtained information aggregation features It contains both global semantic and local texture information, and has undergone... Enhanced processing can provide a foundation for subsequent feature complementarity fusion.

[0044] Step S33: Implementation of the lightweight fusion module; The adaptive complementary features and The image coding features are fused using a lightweight fusion module consisting of depthwise convolutions and ordinary convolutions to obtain the final semantically enhanced features. Specifically: (3); (4); In formulas (3)-(4), These are the initial fusion features generated in the lightweight fusion module. This indicates a channel splicing operation. This indicates a convolution operation.

[0045] Step S4: Based on the semantically enhanced image coding feature pair described in step S3 and A four-dimensional related volume is constructed through pixel-by-pixel dot product operations. Figure 1 middle This represents pixel-by-pixel dot product operation; The specific operation is represented as follows: (20); In formula (20), Represents the four-dimensional related volume, where each location Represents the first frame pixels With the second frame pixels Feature similarity between This represents the vector dot product operation. Indicates the position in the features of the first frame. This indicates the position within the features of the second frame.

[0046] Step S5: Finally, based on the obtained four-dimensional related volume and context features, the output optical flow is iteratively updated and optimized through loop iteration. .

[0047] Specifically, the four-dimensional related volume Contextual features and initial optical flow The input loop update module is used to iteratively update the predicted optical flow, and finally the updated predicted optical flow is upsampled to obtain the original resolution optical flow map. Preferably, the cyclic update module is a GRU-based cyclic update module, which is a commonly used iterative optimization structure in existing optical flow estimation methods. It performs multiple iterative updates on the current optical flow while maintaining the spatial resolution. In each iteration, the current predicted optical flow (initial optical flow in the first iteration), the matching information extracted from the four-dimensional related volume, and the context features are used as inputs. The hidden state is updated by a convolutional gated recurrent unit (GRU), and the optical flow residual is predicted based on the updated hidden state. The optical flow residual is added to the current predicted optical flow to obtain the optical flow prediction result for the next iteration. This cyclic update process is repeated a preset number of times to gradually refine the optical flow estimation results, and finally outputs the optical flow F.

[0048] In summary, this invention innovatively introduces global semantic features from the pre-trained visual model DINOv3 into the optical flow estimation framework and combines it with an adaptive feature fusion module based on the Mamba structure to achieve efficient fusion of local texture features and global semantic features. This approach not only improves the model's matching stability in low-texture, repetitive texture, and occluded regions but also enhances the generalization performance of optical flow estimation in complex and unknown scenarios. Compared with existing methods, this invention achieves better accuracy and robustness while maintaining inference efficiency, demonstrating good engineering applicability and promotional value. Example 2 This embodiment provides a system for implementing an optical flow estimation method that integrates knowledge from Mamba and visual fundamental models. The apparatus includes: At least one processor; and A memory that stores instructions that, when executed by the at least one processor, cause the at least one processor to perform the optical flow estimation method as described above, which integrates knowledge of Mamba and the visual fundamental model.

[0049] In this embodiment, electronic devices include, but are not limited to: personal computers, server computers, workstations, desktop computers, laptop computers, notebook computers, mobile computing devices, smartphones, tablet computers, cellular phones, personal digital assistants (PDAs), handheld devices, messaging devices, wearable computing devices, consumer electronic devices, etc.

[0050] Example 3 This embodiment also provides a computer-readable storage medium storing executable instructions that, when executed, cause the machine to perform the optical flow estimation method described above, which integrates knowledge from Mamba and the visual fundamental model.

[0051] Specifically, a system or apparatus equipped with a readable storage medium may be provided, on which software program code implementing the functions of any of the embodiments described above is stored, and the computer or processor of the system or apparatus can read and execute the instructions stored in the readable storage medium.

[0052] In this case, the program code itself, which can be read from the readable medium, can perform the functions of any of the above embodiments, and therefore the computer-readable code and the readable storage medium storing the computer-readable code constitute a part of this specification.

[0053] Examples of readable storage media include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD-RW), magnetic tapes, non-volatile memory cards, and ROMs. Alternatively, program code can be downloaded from a server computer or the cloud via a communication network.

[0054] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0055] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0056] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0057] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0058] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the technical solutions of the present invention, and are not intended to limit the specific implementation of the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the claims of the present invention should be included within the protection scope of the claims of the present invention.

Claims

1. A method of optical flow estimation fusing Mamba with visual based model knowledge, characterized in that, The method comprises: Step S1, performing feature extraction on input adjacent two frames of images by a convolutional neural network to obtain local texture features of the adjacent two frames of images; and performing down-sampling on a first frame of image of the adjacent two frames of images to obtain context features; Step S2, using a parameter frozen pre-trained visual model to extract global semantic features of the adjacent two frames of images respectively; Step S3, adaptively fusing and enhancing the local texture features and the global semantic features of the adjacent two frames of images respectively to obtain image coding features of the adjacent two frames of images after semantic enhancement; Step S4, constructing a four-dimensional correlation volume by performing pixel-by-pixel multiplication operation on the image coding features of the adjacent two frames of images after semantic enhancement; Step S5, finally, based on the obtained four-dimensional correlation volume and context features, iteratively updating and iteratively optimizing the output optical flow.

2. The method of estimating optical flow fusing Mamba with visual based model knowledge according to claim 1, characterized in that, The step S3 specifically comprises: Step S31, first, the global semantic features of the two adjacent frames of images are respectively preprocessed to align the global semantic features of the two adjacent frames of images with the local texture features of the two adjacent frames of images ; Step S32, then, the aligned global semantic features with local texture features are adaptively fused to obtain information aggregation features ; Then the information aggregation features are obtained by multiplying the global semantic features and the local texture features respectively after the activation of the SILU and ; By introducing a state space structure, global semantic features and local texture features are modeled with long-range dependencies and interact with the information interaction features and are added to obtain adaptive complementary features and , realizing dynamic information transmission across modalities, specifically: （1）； （2）； In formulas (1)-(2), denotes a pixel-wise addition operation, denotes a pixel-wise addition operation, complete model; Step S33, the adaptive complementary feature With The adjacent two frame image final semantic enhanced image coding features are obtained by fusing through the lightweight fusion module composed of deep convolution and ordinary convolution Specifically, （3）； （4）； In formulas (3)-(4), to lightweight fusion module intermediate generated preliminary fusion features, denotes a channel concatenation operation, denotes a convolution operation.

3. The method of estimating optical flow fusing Mamba with visual based model knowledge according to claim 2, characterized in that, The adaptive semantic texture fusion specifically comprises: The global semantic features of two adjacent frames With local texture features Perform element-by-element addition, then... Layer normalization, followed by depthwise convolution activation and Module processing to obtain information aggregation features The specific formula is as follows: （5）； In equation (5), denotes layer normalization operation, denotes a 2D selective scan operation, denotes a depth convolution, denotes an activation function, denotes a linear layer, represents an information aggregation feature.

4. The method of estimating optical flow fusing Mamba with visual based model knowledge according to claim 3, characterized in that, The The processing procedure of the module is: Token embedding is performed on the input features, and four feature input sequences are obtained according to different scanning directions: （6）； In equation (6), denotes the input features, denotes the feature token embeddings, represent the four different directions of the sequence to be scanned; The four different direction to-be-scanned sequences are updated by the S6 scanning module, and an updated scanning sequence is obtained ; The obtained updated scan sequence After sequence aggregation, the information aggregation feature is obtained The obtained information aggregation feature Simultaneously contains global semantic and local texture information, and is enhanced and can provide a basis for subsequent complementary fusion of features.

5. The method of estimating optical flow fusing Mamba with visual based model knowledge according to claim 4, characterized in that, Based on the obtained four-dimensional correlation volume and context features, the output optical flow is iteratively updated and iteratively optimized, specifically: In each iteration process, the current predicted optical flow, the matching information extracted from the four-dimensional correlation volume, and the context features are jointly used as inputs, the hidden state is updated through a convolutional gated recurrent unit GRU, and the optical flow residual is predicted based on the updated hidden state; the optical flow residual is added to the current predicted optical flow to obtain the optical flow prediction result of the next iteration; The loop updating process is repeated for a preset number of times, so as to gradually refine the optical flow estimation result, and finally output the optical flow F.

6. The method of estimating optical flow fusing Mamba with visual based model knowledge according to claim 2, characterized in that, The The state space structure includes two branches: one branch is , , , and , and the other branch is and , specifically: （7）； In equation (7), represents the input of the module, represents the output.

7. The method of estimating optical flow fusing Mamba with visual based model knowledge according to claim 2, characterized in that, The global semantic features are obtained by freezing the parameters of the visual model DINOv3, specifically: The input image, i.e. The encoder extracts respective global semantic features and Specifically, （8）； In formula (8), denotes an input image, denotes a normalization operation, denotes a visual model loaded with pre-trained weights, output denotes global semantic features, comprises and .

8. The method of estimating optical flow fusing Mamba with visual based model knowledge according to claim 2, characterized in that, aligning the global semantic features of the two adjacent frames of images with the local texture features, in particular: Global semantic features are up-sampled to the same resolution as local texture features by bilinear interpolation and then 1 x 1 convolution is applied for channel dimension alignment.​ 9. A system for implementing a method of optical flow estimation that fuses Mamba and visual ground model knowledge, characterized in that, The system comprises: A processor; A memory having a computer program stored thereon and executable on the processor; When the computer program is executed by the processor, the steps of the optical flow estimation method of fusing Mamba and visual basic model knowledge are realized.

10. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 9. When the computer program is executed by the processor, the steps of the method according to any one of claims 1-8 are realized.

Images