Monocular scene flow estimation method and system based on large model coding features, and storage medium

Abstract

The invention belongs to the field of computer vision, and discloses a monocular scene flow estimation method based on large model coding features, a program, equipment and a storage medium. A DAM-MSF model for monocular scene flow estimation is specifically designed, and an image matching feature extraction network driven by a large model, a context feature extraction network and an iterative update network composed of motion update units are designed. The feature extraction network makes full use of universal visual representation learned by a visual basis large model on large-scale data, and effective enhancement of specific tasks by transfer learning is realized. The motion updating unit adopts a large-kernel depth separable convolution module based on an inverted bottleneck structure, the spatial modeling and feature expression ability is enhanced, and the training stability is improved. According to the method, the overall performance and stability of the model can be remarkably improved under the condition that the annotation data of the monocular scene flow task is limited or the training samples are insufficient by effectively utilizing the characteristics with strong generalization ability, and finally, the monocular scene flow estimation precision is improved.

Description

Technical Field

[0001] This invention belongs to the field of computer vision, specifically relating to a monocular scene flow estimation method, system, and storage medium based on large model coding features. Background Technology

[0002] Monocular scene flow estimation aims to recover the 3D motion information of each pixel in a scene from a continuous sequence of images acquired by a monocular camera, including dynamic changes in optical flow and depth. It can be viewed as an extension of 2D optical flow estimation in 3D space. This technology has significant application value in fields such as autonomous driving, intelligent robot perception, and dynamic target analysis. Traditional scene flow estimation methods are mostly based on energy minimization and geometric optimization frameworks, inferring 3D motion information through integrated modeling of optical flow and disparity. However, these methods often suffer from decreased estimation accuracy, high computational cost, and strong reliance on prior knowledge in scenes with complex textures, varying lighting, and large displacements. In recent years, the development of deep learning has significantly improved the performance of scene flow estimation. Deep learning-based models achieve more accurate modeling of motion and depth through recursive update mechanisms, self-supervised learning strategies, and the introduction of geometric consistency constraints. Currently, RAFT-MSF is a representative monocular scene flow method based on convolutional neural networks. RAFT-MSF maps input images to dense feature maps using convolutional neural networks. However, current methods typically rely on handcrafted features or low-dimensional features extracted by small-scale networks. These features lack the ability to model the semantics and structure of complex scenes and are prone to failure in scenes with sparse textures, repetitive backgrounds, or dense dynamic objects. Without a pre-trained large model to provide a general representation, the model must learn from scratch using task-specific data, making it highly sensitive to the amount and quality of data. When training samples are insufficient, expected performance is difficult to achieve, and overfitting is likely to occur. Summary of the Invention

[0003] The purpose of this invention is to provide a monocular scene flow estimation method, system, and storage medium based on large model encoding features. It introduces deep semantic encoding features from a pre-trained large model, fully utilizing its general visual representations learned on large-scale data to achieve effective enhancement of specific tasks through transfer learning. With the help of these features possessing strong generalization capabilities, even in situations where labeled data or training samples are limited for monocular scene flow tasks, the overall performance and stability of the model can still be significantly improved.

[0004] The objective of this invention is achieved through the following technical solution:

[0005] A monocular scene flow estimation method based on large model encoded features, the specific steps of which are as follows:

[0006] Step 1: Acquire images from the monocular camera in adjacent... and Two frames of images at a given time and ;

[0007] Step 2: Construct a monocular scene flow estimation model (DAM-MSF) based on large model encoded features; including an image matching feature extraction network, a context feature extraction network, a 4D correlation pyramid, a motion feature encoder, and an iterative update network; the image matching feature extraction network extracts features from a large visual model with frozen parameters. and Image features , Parallax of the first frame image Then to and The matching features are obtained by downsampling through a weight-sharing convolutional neural network and fine-tuning the two frames of images. and Context feature extraction network and Concatenating and extracting its contextual features via a convolutional neural network Matching features and A 4D correlation pyramid containing global matching information is obtained by element-wise dot product. The motion feature encoder supports 4D correlation pyramids and parallax. Encode to obtain motion features ,Will Half of the context features are divided equally by channel After concatenation, the input is used to iteratively update the network, incorporating the other half of the contextual features. The initial hidden state is used as the initial hidden state; the hidden state is iteratively updated, and then the updated hidden state is input into the scene flow head and disparity head to output the corresponding scene flow and disparity residual. Finally, the scene flow and disparity residual obtained from each iteration are accumulated to output the currently estimated scene flow. and parallax ;

[0008] Step 3: Train the DAM-MSF model using a self-supervised overall network loss function;

[0009] Step 4: Input two consecutive frames of images into the trained model for testing. The visualization output is the estimated scene flow and the depth map of the first frame.

[0010] Furthermore, in the matching feature extraction network and Downsampling is performed using a weight-sharing convolutional neural network, specifically represented as follows:

[0011]

[0012] in, This is a ResNet18 convolutional neural network pre-trained on ImageNet.

[0013] Furthermore, the context feature extraction network... and The data is concatenated and its contextual features are extracted using a convolutional neural network. Specifically,

[0014]

[0015] in, For a ResNet34 convolutional neural network pre-trained on ImageNet, This represents a channel-based splicing operation.

[0016] Furthermore, the 4D correlation pyramid calculates matching features. and The dot product between them is obtained.

[0017]

[0018] in, For 4D related pyramids, Represents the dot product operation. and Representing images respectively and Pixel index;

[0019] The 4D correlation pyramid is further subjected to multi-level pooling to form a correlation quantity pyramid with multiple levels.

[0020] Furthermore, the motion characteristics The encoding process includes: extracting the relevance features from the 4D relevance pyramid. Current estimated optical flow characteristics Currently estimated disparity features and the scene flow features currently estimated. After being stitched together along the channel dimension, it is then fused and reduced in dimension by a convolutional layer.

[0021] Furthermore, the iterative update network is composed of motion update units, specifically represented as follows:

[0022]

[0023] in, The context features are divided equally by channel. This indicates that the channel is divided equally. for and Intermediate features after splicing;

[0024] Will and The hidden state is iteratively updated by feeding it into an iterative update network consisting of two large-kernel depthwise separable convolutional modules.

[0025]

[0026] in, For the first The hidden state obtained in the next iteration. This is the first large-kernel depth-separable convolutional module. for and The intermediate hidden states are obtained through the first large-kernel depthwise separable convolutional module. This is the second large-kernel depthwise separable convolutional module. For the first The hidden state obtained in the next iteration.

[0027] Furthermore, the large-kernel depthwise separable convolution module introduces a Transformer structure, placing depthwise convolutions at the beginning of the module to replace the self-attention layer in the Transformer structure, and pointwise convolutions to replace the linear layers in the Transformer structure. Specifically,

[0028]

[0029] in, The output features are those obtained after passing through a large kernel depth-separable convolutional module. For the input features fed into the large kernel depthwise separable convolutional module, For depthwise separable convolution, For layer normalization, This is convolutional layer 1. For GELU activation function, This is convolutional layer 2.

[0030] Furthermore, the overall network loss function is,

[0031]

[0032] in, For scene flow loss; This is a deep loss; and These are the weighting coefficients for depth loss and scene flow loss.

[0033] A computer system includes a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement steps of a monocular scene flow estimation method based on large model coded features.

[0034] A computer-readable storage medium having a computer program / instructions stored thereon, which, when executed by a processor, implements the steps of a monocular scene flow estimation method based on large model encoded features.

[0035] The beneficial effects of this invention are as follows:

[0036] This invention leverages the deep semantic encoding features of a pre-trained large-scale model, fully utilizing its general visual representations learned on massive datasets, to effectively enhance transfer learning for specific tasks. With these highly generalizable features, even in monocular scene flow tasks with limited labeled data or insufficient training samples, the overall performance and stability of the model can be significantly improved. The multi-scale, highly expressive global features provided by the large model not only improve the estimation accuracy of traditional methods in scenes with weak texture, occlusion, or varying lighting, but also enable more reliable 3D motion inference capabilities from monocular inputs, thereby expanding the applicability and practical value of monocular scene flow estimation techniques. Attached Figure Description

[0037] Figure 1 This is a flowchart of the present invention;

[0038] Figure 2 Diagram of the DAM-MSF network for monocular scene flow estimation;

[0039] Figure 3 To iteratively update the network CUM structure diagram;

[0040] Figure 4 This is a structural diagram of a large kernel depth separable convolutional module. Detailed Implementation

[0041] The present invention will now be further described with reference to the accompanying drawings.

[0042] Example 1:

[0043] The overall process of this invention is as follows: Figure 1 As shown, this is achieved through the following steps:

[0044] Step 1: Construct monocular image sequences for training and testing the monocular scene flow network, including images from a monocular camera. , Image of time and .

[0045] Step 2: Construct a monocular scene flow estimation model DAM-MSF based on large model encoded features; including an image matching feature extraction network, a context feature extraction network, a 4D correlation pyramid, a motion feature encoder, and an iterative update network;

[0046] 1. Construct a large model-driven image matching feature extraction network, with the first frame image as input. and The output is the matching features. and ;

[0047] like Figure 2 As shown, the large model-driven matching feature extraction network is used to extract feature representations for monocular scene flow matching operations from the two consecutive input images. The two images are input into a parameter-frozen visual foundation large model, which can be any general visual representation model trained on large-scale data, such as DINO, Segment AnythingModel (SAM), and the Depth Anything series models, to obtain stable and generalizable feature descriptions. This example uses Depth Anything v2, which is built on Vision Transformer-Base (ViT-B) and consists of a patch embedding module, a multi-layer Transformer encoder, and a multi-scale feature generation module. First, the input image is processed by the patch embedding module according to... The patch size is used to divide the data, and the resulting data is divided into segments with a size of [missing information]. The convolutional kernel is mapped to an initial token representation of dimension 128, forming Each token is a spatial token. These tokens are then fed into an encoder consisting of 12 Transformer blocks, where global correlation features are extracted using multi-head self-attention and an MLP structure. Finally, the model outputs a 4-level multi-scale 2D feature map and disparity maps with the same resolution as the input image, based on features from different Transformer layers. The number of channels in the two-dimensional feature maps are 96, 192, 384, and 768, respectively, corresponding to resolutions of approximately 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the input image. The topmost feature is taken as the image feature. and Its operation is defined as follows:

[0048]

[0049] in, The frozen visual foundation large model, Depthing Anything v2, and The input consists of two consecutive frames of images, with a size of [size missing]. , and The image features are obtained through a large visual model, with a size of [size missing]. , The first frame disparity predicted using a large visual model has a size of [value missing]. .

[0050] Fine-tuning of image features generated by large models is achieved using a ResNet18 convolutional neural network pre-trained on ImageNet. ResNet18 consists of a large convolutional neural network with a size of [missing information - likely a value]. A convolutional layer with a size of It consists of a max-pooling layer and six residual blocks. It combines the two image features output from the large model. and The features are sequentially fed into ResNet18 for uniform feature compression and downsampling to obtain matching features. and Its operation is defined as follows:

[0051]

[0052] in, For a ResNet18 convolutional neural network pre-trained on ImageNet, and The matching features obtained after fine-tuning have a size of [value missing]. .

[0053] 2. Construct a large model-driven context feature extraction network, with the image features of the first frame as input. The output is context features. ;

[0054] like Figure 2 As shown, the large model-driven context feature extraction network is used to generate the global context information required for scene flow iterative optimization. It fuses features from the large model encoder with deep priors, and then uses a ResNet34 convolutional neural network pre-trained on ImageNet to extract local structures from the fused features, forming an enhanced context representation. The ResNet34 network consists of a large model encoder with a depth prior. A convolutional layer with a size of It consists of a max pooling layer and 13 residual blocks, and its operation is defined as follows:

[0055]

[0056] in, For a ResNet34 convolutional neural network pre-trained on ImageNet, This indicates a channel-based splicing operation. For contextual features, size is .

[0057] 3. The 4D correlation pyramid calculates matching features. and The visual similarity between two images is represented by a complete correlation between them. Given matching features... and The relevant volume is obtained by calculating the dot product between all vector pairs of the two feature maps. The calculation process is as follows:

[0058]

[0059] in, This is a 4D related pyramid, its dimensions are... , Represents the dot product operation. and Representing images respectively and Pixel index.

[0060] 4D Related Pyramid Corresponding to matching features and The resolution size is determined, and then the last two dimensions are pooled using pooling layers with kernel sizes of 1, 2, 4, and 8 to obtain a correlation pyramid with 4 layers. Correlation quantity at each layer The size is .

[0061] The motion feature encoder consists of nine convolutional layers, all with a stride of 1. Different kernel sizes are used to balance local texture capture and global feature fusion. Specifically, the relevance feature branch uses 1×1 and 3×3 convolutions to compress channels and extract matching dependencies; the optical flow branch uses 7×7 and 3×3 convolutions to expand the receptive field and capture large-scale motion; the parallax branch also uses 7×7 and 3×3 convolutions to enhance depth variation feature extraction; and the scene flow branch uses 7×7 and 3×3 convolutions to achieve 3D dynamic feature modeling. After extraction through their respective two-layer convolutions, the four features are concatenated through channel dimensions to form a high-dimensional fusion tensor, which is then processed by a 3×3 convolutional layer to integrate information and reduce dimensionality, generating motion features. Its specific operation definition is as follows:

[0062]

[0063] in, The correlation features are extracted from the 4-dimensional correlation volume through the correlation feature branch, and their size is... , For ReLU activation functions, and Two convolutional layers for the relevance feature branches; The optical flow currently estimated The optical flow features extracted through the optical flow branch have a size of , The current estimated optical flow has a magnitude of , and Two convolutional layers that form the optical flow branch; The disparity obtained from the current estimate The disparity features extracted through the disparity branch have a size of , The current estimated disparity has a magnitude of , and Two convolutional layers for the parallax branch; For the scene flow obtained from the current estimation The scene flow features extracted through the scene flow branch have a size of , The current estimated scene flow has a size of , and Two convolutional layers that form the optical flow branch; Convolutional layers for extracting motion features.

[0064] 4. Construct an iterative update network composed of motion update units. The input is a concatenated feature of motion features and context features divided equally by channel. The other half of the context features is used as the initial hidden state. The hidden state is iteratively updated. The corresponding scene flow and disparity are output through the scene flow head and disparity head.

[0065] The motion update unit can be constructed from models capable of modeling data, such as convolutional GRU, Transformer, and Mamba. Taking a large-kernel depthwise separable convolutional module based on an inverted bottleneck structure as an example, such as... Figure 3 As shown, the iterative update network composed of motion update units adopts a large-kernel depthwise separable convolutional module based on an inverted bottleneck structure instead of the traditional convolutional GRU structure. This module generates new hidden state representations and the basis for subsequent scene flow and disparity predictions in each iteration, based on the current hidden state, context features, and motion features. The context features obtained through the context feature extraction network are then divided into context features according to their channels. and initial hidden state Contextual features Motion features obtained through a motion encoder The splicing operation is defined as follows:

[0066]

[0067] in, The context features are divided equally by channel, with a size of , The initial hidden state has a size of , This indicates that the channel is divided equally. for and The size of the concatenated intermediate feature is The hidden states obtained through the motion update unit are not limited to large kernel depthwise separable convolutional modules using inverted bottleneck structures. Hidden states generated by convolutional neural networks, Transformer, and Mamba are also applicable to this invention and are all within the scope of protection of this invention.

[0068] The spliced ​​intermediate features The initial hidden state is fed into an iterative update network consisting of two large-kernel depthwise separable convolutional modules to iteratively update the hidden state. The operation is defined as follows:

[0069]

[0070] in, For the first The hidden state obtained in the next iteration has a size of , This is the first large-kernel depth-separable convolutional module. This is the second large-kernel depthwise separable convolutional module. For the first The hidden state obtained in the next iteration has a size of ;

[0071] like Figure 4 As shown, the large-kernel depth-separable convolution module improves upon traditional residual structure convolutional neural networks by introducing key Transformer technologies. It significantly enhances spatial modeling and feature representation capabilities through large-kernel depth-separable convolution and an inverted bottleneck structure. Simultaneously, it improves training stability by utilizing layer normalization and the GELU activation function, resulting in stronger parallelism and higher inference efficiency. Its operation is defined as follows:

[0072]

[0073] in, The output features are those obtained after passing through a large kernel depth-separable convolutional module. For the input features fed into the large kernel depthwise separable convolutional module, For size Depth-separable convolution, For layer normalization, For size Convolutional layer 1, For GELU activation function, For size Convolutional layer 2.

[0074] The final hidden state of the iteratively updated network output is then used to output the currently estimated disparity and scene flow through disparity and scene flow heads. Three convolutional layers, all with a size of [size missing], are employed. With a step size of 1, the scene flow and disparity residuals are predicted separately. The network output is iteratively updated. The currently estimated scene flow and disparity are the sum of the scene flow and disparity residuals obtained in each iteration. The specific operation is as follows:

[0075]

[0076]

[0077] in, and For the current number The scene flow and disparity obtained in the next iteration and For the first The scene flow and disparity estimates after the next iteration. and This represents the residual update amount of the scene flow and disparity calculated by the update network in the current iteration step. and These are the scene stream head and the parallax head, respectively.

[0078] Step 3: Input two consecutive frames of images into the network input terminal and perform supervised training on the network using the overall network loss function.

[0079] Overall network loss function It consists of a weighted sum of depth loss and scene flow loss, and its specific definition is as follows:

[0080]

[0081] in, For scene flow loss; This is a deep loss; and The weighting coefficients for depth loss and scene flow loss are typically defined as a constant.

[0082] This network uses disparity information from stereo image pairs to guide depth estimation during the training phase. To achieve more accurate depth estimation, the network employs both photometric loss and smoothing loss methods, with a regularization constant set to 0.1. The specific expression for the depth loss is:

[0083]

[0084] in, To smooth out the loss; For photometric loss; and These are the weighting coefficients for photometric loss and smoothing loss.

[0085] The specific expression for smoothing loss is:

[0086]

[0087] in, This represents the total number of pixels. Represents the pixel coordinates of the image; Indicates horizontal and vertical directions; For depth map in direction The second-order gradient measures the rate of change of depth. For the original image, This represents the image gradient, reflecting edge strength. The attenuation parameter in the exponent term controls the edge constraint strength. When the edge strength is large (such as object boundaries), the exponent value of this term is small, thereby reducing the smoothing constraint at the boundary.

[0088] The specific expression for photometric loss is:

[0089]

[0090] in, The input is the original left view image; A left-view image of the target reconstructed through parallax; For parallax occlusion masking; The loss function is for occlusion perception.

[0091] Occlusion perception loss function The Hamming distance for calculating visible pixels is expressed as follows:

[0092]

[0093] in, and These are the reference image and the reconstructed image, respectively. This is a weight mask used to mark the valid pixel region; For image In pixels The structural descriptor of the location and its neighborhood is used to describe the local intensity difference and is standardized to avoid excessively large values; This is a smoothing term to prevent numerical instability caused by a denominator of zero. Occlusion-aware loss function. The overall idea is to use pixels The surrounding neighborhood computes the local structure descriptor. The structural similarity between the reference image and the reconstructed image is compared. A weighted mask is used in the denominator. The effective region is weighted and summed, and finally constrained by normalization.

[0094] The scene flow loss used in this network consists of photometric loss, 3D reconstruction loss, and smoothing loss, and its specific expression is as follows:

[0095]

[0096] in, This is due to the loss of scene rasterism; This represents the loss during 3D reconstruction. To smooth out the loss; , and Each represents a loss weighting coefficient.

[0097] The specific expression for scene moiré loss is:

[0098]

[0099] in, The original image as input; The target image reconstructed from the scene stream; Use a scene flow occlusion mask; The occlusion perception loss function is defined. First, the camera's internal matrix is ​​used. and estimated depth and scene flow Obtain the synthetic reference image Then, given a reference image and synthetic reference image Compensate for scene flow smoothing loss. Scene flow occlusion mask. By back-view scene flow The result is obtained after de-occlusion. Finally, the occlusion-aware loss function is used. Calculate the scene rheometry loss.

[0100] The 3D reconstruction loss is calculated by applying Euclidean distance to the 3D points of the visible pixels. The specific expression is as follows:

[0101]

[0102] in, and for Time and Depth map; This is the inverse of the camera intrinsic parameter matrix; For scene flow in pixels The loss function calculates the 3D displacement vector at a given location. It constrains the consistency of depth with scene flow by calculating the 3D Euclidean distance between corresponding pixels in adjacent frames. To eliminate scale effects, depth is normalized using the camera focal length and stereo baseline.

[0103] The specific expression for smoothing loss is:

[0104]

[0105] in, For scene flow In direction The second-order gradient is used to measure the smoothness of the scene flow in that direction; For image-based The weights of the gradient. Indicates the edge intensity of objects in an image; This is a depth-based 3D distance normalization term used to normalize the smoothing loss of the scene flow according to object distances, ensuring consistent smoothness in 3D space.

[0106] Step 4: Input two consecutive frames of images into the trained model for testing, and visualize the corresponding scene flow and the depth map of the first frame.

[0107] In particular, in some preferred embodiments of the present invention, a computer device is also provided, including a memory and a processor and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the monocular scene flow estimation method based on large model coding features described in any of the above embodiments.

[0108] In some other preferred embodiments of the present invention, a computer-readable storage medium is also provided, on which a computer program / instruction is stored, wherein when the computer program is executed by a processor, it implements the steps of the monocular scene flow estimation method based on large model coding features described in any of the above embodiments.

[0109] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the monocular scene flow estimation method based on large model coding features as described above. The process of the embodiments will not be repeated here.

[0110] Computer-readable storage media encompass a variety of types, including persistent and non-persistent, portable and fixed. These media store information using different technologies, and the content can be machine instructions, data structures, program modules, or other types of data. Some typical examples of computer storage media include: phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), various types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory and other storage technologies, optical storage media such as CD-ROM and digital video disc (DVD), magnetic storage devices such as magnetic tape and disks, and other non-transferable media used to store information accessible to computing devices. It is important to note that the computer-readable media described herein do not include temporary storage media, such as modulated data signals and carrier waves.

[0111] Those skilled in the art will further recognize that the operation of the module can be achieved using existing technical protocols or programs, without relying on new computer programs themselves. The units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0112] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented in hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0113] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A monocular scene flow estimation method based on large model encoded features, characterized in that: The specific steps are as follows: Step 1: Acquire images from the monocular camera in adjacent... and Two frames of images at a given time and ; Step 2: Construct a monocular scene flow estimation model (DAM-MSF) based on large model encoded features; including an image matching feature extraction network, a context feature extraction network, a 4D correlation pyramid, a motion feature encoder, and an iterative update network; the image matching feature extraction network extracts features from a large visual model with frozen parameters. and Image features , Parallax of the first frame image Then to and The matching features are obtained by downsampling through a weight-sharing convolutional neural network and fine-tuning the two frames of images. and ; Context feature extraction network and Concatenating and extracting its contextual features via a convolutional neural network ; Matching features and A 4D correlation pyramid containing global matching information is obtained by element-wise dot product. ; Motion feature encoders for 4D correlation pyramids, parallax Encode to obtain motion features ,Will Half of the context features are divided equally by channel After concatenation, the input is used to iteratively update the network, incorporating the other half of the contextual features. As the initial hidden state; The hidden state is iteratively updated, and then the updated hidden state is input into the scene flow head and disparity head to output the corresponding scene flow and disparity residual. Finally, the scene flow and disparity residual obtained from each iteration are accumulated to output the currently estimated scene flow. and parallax ; Step 3: Train the DAM-MSF model using a self-supervised overall network loss function; Step 4: Input two consecutive frames of images into the trained model for testing. The visualization output is the estimated scene flow and the depth map of the first frame.

2. The monocular scene flow estimation method based on large model coding features according to claim 1, characterized in that: In the matching feature extraction network and Downsampling is performed using a weight-sharing convolutional neural network, specifically represented as follows: in, This is a ResNet18 convolutional neural network pre-trained on ImageNet.

3. The monocular scene flow estimation method based on large model coding features according to claim 1, characterized in that: The context feature extraction network and The data is concatenated and its contextual features are extracted using a convolutional neural network. Specifically, in, For a ResNet34 convolutional neural network pre-trained on ImageNet, This represents a channel-based splicing operation.

4. The monocular scene flow estimation method based on large model coding features according to claim 1, characterized in that: The 4D correlation pyramid calculates matching features. and The dot product between them is obtained. in, For 4D related pyramids, Represents the dot product operation. and Representing images respectively and Pixel index; The 4D correlation pyramid is further subjected to multi-level pooling to form a correlation quantity pyramid with multiple levels.

5. The monocular scene flow estimation method based on large model coding features according to claim 1, characterized in that: The motion characteristics The encoding process includes: extracting the relevance features from the 4D relevance pyramid. Current estimated optical flow characteristics Currently estimated disparity features and the scene flow features currently estimated. After being stitched together along the channel dimension, it is then fused and reduced in dimension by a convolutional layer.

6. The monocular scene flow estimation method based on large model coding features according to claim 1, characterized in that: The iterative update network consists of motion update units, specifically represented as follows: in, The context features are divided equally by channel. This indicates that the channel is divided equally. for and Intermediate features after splicing; Will and The hidden state is iteratively updated by feeding it into an iterative update network consisting of two deep separable convolutional modules with large kernels. in, For the first The hidden state obtained in the next iteration. This is the first large-kernel depth-separable convolutional module. for and The intermediate hidden states are obtained through the first large-kernel depthwise separable convolutional module. This is the second large-kernel depthwise separable convolutional module. For the first The hidden state obtained in the next iteration.

7. The monocular scene flow estimation method based on large model coding features according to claim 6, characterized in that: The large-kernel depthwise separable convolution module introduces a Transformer structure, placing depthwise convolutions at the beginning of the module to replace the self-attention layer in the Transformer structure, and pointwise convolutions to replace the linear layers in the Transformer structure. Specifically... in, The output features are those obtained after passing through a large kernel depth-separable convolutional module. For the input features fed into the large kernel depthwise separable convolutional module, For depthwise separable convolution, For layer normalization, This is convolutional layer 1. For GELU activation function, This is convolutional layer 2.

8. The monocular scene flow estimation method based on large model coding features according to claim 1, characterized in that: The overall network loss function is: in, For scene flow loss; This is a deep loss; and These are the weighting coefficients for depth loss and scene flow loss.

9. A computer system comprising a memory, a processor, and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that: When executed by a processor, the computer program implements the steps of the method according to any one of claims 1 to 8.

Images