Monocular depth estimation method and device based on bidirectional state space model, and medium

Abstract

The invention discloses a monocular depth estimation method and device based on a bidirectional state space model, and a medium, relates to an image processing technology, and solves the problems that an existing self-supervision method is insufficient in the aspects of long sequence modeling, training speed, reasoning speed and scene complex detail capturing, and the optimization effect of the model under sparse supervision is poor. The monocular depth estimation method comprises the following steps: constructing an encoder-decoder structure which comprises a continuous expansion convolution module and a bidirectional state space module, generating intermediate features of a training image by an encoder, modeling long-distance intermediate feature information, and generating output features by a decoder; constructing an attitude estimation model, performing joint reconstruction by using output features generated by a decoder and depth estimation of a training image, modeling a learning target, and designing a loss function of a target image and a reconstructed image to optimize a network; and training the constructed model by adopting a public KITTI data set to obtain a weight model, and evaluating the precision of monocular image depth estimation by the trained model. The method is used for image monocular depth estimation.

Description

Technical Field

[0001] The present invention relates to an image processing technology, in particular to a monocular depth estimation method, device and medium based on a bidirectional state space model. Background Art

[0002] At present, monocular visual depth estimation aims to use videos or pictures taken by a single camera to infer the distance or depth information of objects in the scene. This technology is crucial for fields such as robotics and autonomous driving. In recent years, although deep learning technology has made rapid progress in the field of computer vision, depth estimation technology based on the encoder-decoder architecture still faces many challenges.

[0003] Traditional visual depth estimation methods usually rely on binocular cameras or RGBD cameras to obtain depth information. However, with the development of deep learning technology, monocular depth estimation algorithms based on deep learning models have been widely used. These methods usually use supervised learning to train and optimize models through a large amount of data with depth labels. However, obtaining such a large amount of depth-labeled data faces huge challenges in practical applications, which leads to certain limitations in the generalization ability of these methods.

[0004] To solve this problem, self-supervised learning methods have gradually become a research hotspot. Through the "encoder-decoder" structure, these methods can use their own data to design optimization goals without the need for real deep labels, breaking through the limitations of traditional supervised learning methods.

[0005] For example, document number CN115731280A discloses a self-supervised monocular depth estimation method based on Swin-Transformer and CNN parallel network. The invention uses Swin-Transformer and CNN for feature extraction at the same time, and fuses the extracted features, so that the network can strike a balance between establishing long-range correlation and retaining spatial structure information, enhance the network's ability to learn features, and combine the scale-by-scale self-distillation loss proposed in the present invention to perform self-supervised training of the network, thereby improving the accuracy of self-supervised monocular depth estimation.

[0006] The prior art with the document number CN116883476A discloses a monocular depth estimation method based on attention feature fusion and multi-level correction (CN202310783026.7), which includes obtaining and preprocessing the data set of the image and the corresponding real depth image; pre-training the constructed ST coding network structure on the image classification data set; processing the image in blocks and extracting features; inputting the coding features into the monocular depth estimation network model, using the attention feature fusion module to perform progressive fusion of adjacent resolution features; processing the correction network of the decoded features, and correcting the decoded features; summing using the loss function; continuously adjusting the parameters of the monocular depth estimation network model through the optimizer; and obtaining the depth value of each pixel. Accurate prediction of local detail information of the depth image is achieved.

[0007] However, current self-supervised methods have shortcomings in long sequence modeling, training speed, inference speed, and capturing complex details of scenes, and the optimization effect of the model under sparse supervision also needs to be improved. Summary of the invention

[0008] The technical problem to be solved by the present invention is: to solve the problem that the current self-supervision method is insufficient in long sequence modeling, training speed, inference speed and capturing complex details of the scene, and the optimization effect of the model under sparse supervision is not good.

[0009] The present invention is implemented by the following technical solution: a monocular depth estimation method based on a bidirectional state space model is provided, comprising the following steps:

[0010] Step S1: construct an encoder-decoder structure, including a continuous dilated convolution module and a bidirectional state space module, the encoder generates intermediate features of the training image, models the long-distance intermediate feature information, and the decoder generates output features;

[0011] Step S2: Build a pose estimation model, and use the output features generated by the decoder and the depth estimation of the training image to jointly reconstruct the learning target, design the loss function of the target image and the reconstructed image to optimize the network;

[0012] Step S3: Use the public KITTI dataset to train the constructed model to obtain a weight model, and evaluate the accuracy of the trained model for monocular image depth estimation.

[0013] Compared with the prior art, the present invention has the following beneficial effects:

[0014] 1. The monocular depth estimation method based on the bidirectional state-space model can effectively model long sequences of images. At the same time, this method does not generate secondary computational complexity related to the attention mechanism, thereby enhancing the model's ability to estimate the depth of monocular images and extract fine-grained local features more finely. At the same time, it also reduces the difficulty of model training and improves the model's reasoning speed, thereby solving the problem of poor performance of monocular depth estimation of images. Compared with the existing technology, the accuracy of monocular depth estimation of images is improved.

[0015] 2. The method of the present invention can effectively model long sequences of images, has fast training and reasoning speeds, accurately captures complex details of scenes, and significantly improves the optimization effect under sparse supervision. The present invention solves the problems of slow training speed, insufficient performance, and capturing complex details of scenes in monocular depth estimation models, improves the generalization ability of the model, and effectively reduces the difficulty of industrial deployment of monocular depth estimation methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 is a flow chart of the present invention;

[0017] Figure 2 Network structure diagram of monocular depth estimation method based on bidirectional state space model;

[0018] Figure 3 Schematic diagram of the continuous dilation convolution module;

[0019] Figure 4 Schematic diagram of the bidirectional state space module;

[0020] Figure 5 This is a diagram of the visualization results on the KITTI dataset. DETAILED DESCRIPTION

[0021] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0022] See attached Figure 1 , this embodiment provides a monocular depth estimation method based on a bidirectional state space model, comprising the following steps:

[0023] Step S1: construct an encoder-decoder structure, including a continuous dilated convolution module and a bidirectional state space module, the encoder generates intermediate features of the training image, models the long-distance intermediate feature information, and the decoder generates output features;

[0024] The step S1 constructs an encoder-decoder structure, wherein the encoder generates intermediate features of the training image, models the long-distance intermediate feature information, and the decoder generates output features, including:

[0025] The backbone of the encoder is constructed, wherein the backbone of the encoder includes a convolution module (Conv Stem) and an intermediate feature extraction layer, wherein the intermediate extraction layer includes a continuously dilated convolution module and a bidirectional state space module; after a training image is processed by the convolution module (Conv Stem), a feature matrix that can be processed by the bidirectional state space module is generated by the continuously dilated convolution module; the output feature matrix is ​​input to the bidirectional state space module for processing to model long-distance intermediate feature information; based on the feature matrix output by the encoder, depth information of the training image is generated by decoding through a three-layer decoder.

[0026] See attached Figure 2 In this embodiment, the size and channel information of the training image are processed to obtain features of shape [B, C, H, W], which are passed through the convolution module and the average pooling module respectively. After the convolution layer, the training image is passed through the average pooling layer to average the image area, and the feature vector output by the convolution layer is reduced in dimension, thereby reducing information redundancy, improving the training speed of the model, and improving the robustness of the extracted features. Among them, B represents batch size, C represents the number of channels, H represents height, and W represents width.

[0027] Input training image [B, C, H in ,W in ], first, the output features of the convolution module and the average pooling module enter the intermediate feature extraction layer. The intermediate extraction layer consists of a continuous expansion convolution module and a bidirectional state space module.

[0028] The shape is [B,C,H in ,W in ] The training image is input into the average pooling module, and the output is [B, C, H out ,W out ], the average pooling module can be expressed as:

[0029]

[0030] The stride represents the window movement step, which is the same as kernel size Keep it consistent and round down to get [B,C,H out ,W out ]output.

[0031] For details, see the attached Figure 3 , extract the image processed by the convolution module (Conv Stem) through depth-wise separable dilated convolution to generate a feature matrix with multi-scale local information, the steps include:

[0032] The image processed by the convolution module (Conv Stem) is extracted through depth-wise separable dilated convolution to generate a feature matrix with multi-scale local information; the batch normalization layer (BatchNorm) vertically normalizes the multi-scale local features to obtain normalized features; the normalized features generate a new feature matrix (FeatureMap) through a one-dimensional convolution kernel; the new feature matrix is ​​activated by a RELU activation function; the features activated by the RELU activation function are passed through a one-dimensional convolution kernel to generate a feature matrix that can be processed by a bidirectional state space module.

[0033] See attached Figure 4 The output feature matrix is ​​input to the bidirectional state space module for processing, and the steps of modeling the long-distance intermediate feature information include:

[0034] Accept the continuous expansion convolution module to generate a dimension of [B, C, H in ,W in ] feature matrix, where B is the batch size, C is the number of channels, H is the height of the feature map, and W is the width of the feature map. The feature map is processed by the convolution module to obtain the feature matrix output by the convolution layer; the feature matrix output by the convolution layer is respectively generated by the fully connected layer 1 and the fully connected layer 2 to generate feature branch 1 (x1) and feature branch 2 (x2), and the ReLU6 activation function is applied to branch x1 and multiplied by x2 to obtain a shape of [B, C, H in ,W in ]’s fusion feature matrix;

[0035] The fused feature matrix is ​​further processed by deep convolution (without batch normalization) to obtain a new feature with a shape of [B, C, H, W]. The new feature map with a shape of [B, C, H, W] is dimensionally transformed, and the matrix shape changes from [B, C, H, W] to [B, H*W, C] to obtain a feature matrix with a shape of [B, H*W, C]. The feature matrix with a shape of [B, B*W, C] is input into the projection layer of the bidirectional state space module, and the received input features are linearly projected, and the feature dimension is expanded to twice the original dimension for subsequent feature processing. The features output by the projection layer are received, and a one-dimensional convolution kernel is applied to perform a convolution operation to capture local spatiotemporal dependencies.

[0036] The feature sequence after one-dimensional convolution is activated by the SiLU activation function; the activated feature sequence is projected to the state space model, including the time step (dt), the state matrix B and the state matrix C, which are used to update various parameters in the state space, including the bidirectional state space matrix, the convolution block and the fully connected layer, which are used to process the input features from both the forward and reverse directions, model the long-distance intermediate feature information, and obtain the bidirectional state space model output; the input features are directly added to the bidirectional state space model output to form a residual connection to ensure stable gradient propagation; the output features of the bidirectional state space model are reprojected to the input dimension, and the output feature matrix shape is [B, C, H, W].

[0037] The batch normalization layer (BatchNorm) vertically normalizes the multi-scale local features to obtain normalized features. The specific steps are as follows:

[0038] a) Save the input X for residual connection operation;

[0039] b)DDWConv r (·) is the dilated convolutional layer, which can expand the receptive field without increasing the number of parameters. The output dimension remains unchanged, which is still [B, C, H, W];

[0040] c) The batch normalization layer (BatchNorm) vertically normalizes the multi-scale local features to obtain normalized features, which can be expressed as:

[0041]

[0042] in, is the sample mean of the mini-batch sample B, is the sample standard deviation of sample B. After applying normalization, the mean of the generated mini-batch is 0 and the unit variance is 1. In addition, the scaling parameter γ and the offset parameter β are defined, and the shape is the same as the input parameter x;

[0043] d)Linear G1 For the point-by-point convolution layer 1, the feature dimension of each pixel position is expanded from C to 6C, which can improve the feature representation capability;

[0044] e) ReLU is the activation function, which can be expressed as ReLU(x)=max(0,x);

[0045] f)Linear G2 For the point-by-point convolution layer 2, the feature dimension of each pixel is restored from 6C to C, and the initial number of channels is restored;

[0046] g) Perform residual connection, add the input X to the regularized output X to form a skip connection, and finally get the output of the continuous expansion convolution module The shapes are [B,C,H,W] and [B,C,H in ,W in ]Stay consistent.

[0047] High-dimensional and nonlinear feature transformation is crucial in deep learning. This necessity stems from the complexity of real-world monocular image depth data and the ability of the model to represent such data. Usually, low-dimensional features are first mapped to high-dimensional space through linear projection, and then nonlinear activation functions are used to introduce nonlinear structures to better express the complexity of the data.

[0048] For example, the polynomial kernel function can be expressed as k(x1,x2)=(γx1x2+c) d The input features can be Projection to (n+1) d In the dimensional space, the Gaussian kernel function can transform k(x1,x2) into Projected into infinite-dimensional feature space.

[0049] This work uses a novel operation to obtain high-dimensional and nonlinear space from low-dimensional input features. This operation has the unique ability to implicitly consider extremely high-dimensional features while performing calculations in low-dimensional space.

[0050] This significant advantage makes star operation different from other technologies in the field of efficient networks and suitable for efficient network design. The specific process is as follows:

[0051] In a single-layer neural network, the new operation is represented by The features of two linear transformations are fused by element-wise multiplication, let And the new operation can be simplified to

[0052] definition represents a set of C+1-dimensional real vectors, where C is the number of channels of the input feature. The star operation can be expressed as the following equation:

[0053]

[0054] in:

[0055]

[0056] Extended to a 3-layer network structure, as follows:

[0057]

[0058] This method can effectively and reasonably amplify the implicit dimension of the input features, allowing the network to better learn effective features. At the same time, it only uses a three-layer network, greatly saving the resources consumed by model training. At the same time, it solves the problem of poor depth estimation of monocular images and poor depth estimation performance of the model.

[0059] Accepts features of dimension [B, C, H, W] generated by continuous dilated convolution modules The features are processed through the convolution module to obtain the feature matrix output by the convolution layer;

[0060] The feature matrix output by the convolutional layer generates two feature branches x1 and x2 respectively through the fully connected layer 1 and the fully connected layer 2. The ReLU6 activation function is applied to branch x1 and multiplied with x2 to obtain a fused feature matrix with a shape of [B, C, H, W].

[0061] The fused feature matrix is ​​further processed by deep convolution (without batch normalization) to obtain a new feature with the shape of [B, C, H, W]

[0062] Will The dimensions are resized to [B, C, H*W], that is, the height and width are flattened and merged into one dimension H*W, and then the feature tensor dimensions are rearranged, [B, C, H*W] is rearranged to [B, H*W, C], and the new The shape is [B,H*W,C], which is convenient for passing to the selected state space model for processing.

[0063] After adjusting the dimensions Projecting to the fully connected layer to obtain features B, U and △ can be expressed as:

[0064]

[0065] The state space model parameter matrix is ​​obtained by calculating the characteristics B, C and △ It can be expressed as:

[0066]

[0067] For the state-space model parameter matrix And feature U to model the state space model, output That is, the input feature The output features after long sequence modeling by the state space model (SSM) can be expressed as:

[0068]

[0069] These vectors are processed in both forward and backward directions to simulate the behavior of a bidirectional neural network layer. In each direction, a one-dimensional convolution module and a Sigmoid Linear Unit (SiLU) activation function are applied to capture the contextual information of image features.

[0070] When processing image block sequences, the state-space sequence model significantly improves its ability to capture the global contextual information of the image through a bidirectional information transmission mechanism. Specifically, this model utilizes the information flow in both forward and backward directions to comprehensively integrate the information in the image.

[0071] In the bidirectional information transmission mechanism, in the forward process, the model starts from the starting position of the image sequence, gradually processes each feature image, and gradually transmits and integrates information. This processing method allows the model to form a comprehensive understanding of the current image block based on the content of the current feature image and the previously processed information; in the backward information transmission process, the model processes the image block forward from the end of the sequence. This process can integrate the information from the end of the sequence to the current position, so that the model can use the upcoming image context information to further enrich the representation of the current image block. This global perspective enables the model to capture the complex relationships in the image, including the relative positions of objects and background information.

[0072] The bidirectional processing method enables the model to fuse information from different directions of the training images, enhances the ability to model the dependencies between data, realizes long sequence modeling, and effectively improves the accuracy of the model in monocular depth estimation. At the same time, the model can also show stronger stability when processing complex visual scenes.

[0073] The activated feature sequence is projected into the state space model, including the time step (dt), state matrix B and state matrix U, which are used to update various parameters in the state space, including the bidirectional state space matrix, convolutional blocks and fully connected layers, which are used to process input features from both the forward and reverse directions, model long-distance intermediate feature information, and obtain the bidirectional state space model output;

[0074] The input features of the bidirectional state-space module and the output features after passing through the fully connected layer in the module are directly added to the bidirectional state-space model output to form a residual connection to ensure stable gradient propagation.

[0075] Step S2: Build a pose estimation model, and use the output features generated by the decoder and the depth estimation of the training image to jointly reconstruct the learning target, design the loss function of the target image and the reconstructed image to optimize the network;

[0076] The steps of constructing a posture estimation model in step S2, and jointly reconstructing the depth estimation of the training image using the output features generated by the decoder, modeling the learning target, and designing the loss function of the target image and the reconstructed image to optimize the network include:

[0077] A pose estimation model (PoseNet) is constructed. The model inputs two adjacent frames in the training data set and outputs the relative pose between the two frames. The network extracts high-dimensional features from the input RGB image; the high-dimensional features generate feature maps through the feature extraction layer, and the feature maps are processed through convolution operations to extract information about the camera motion in the image; the model finally outputs a pose vector containing six-dimensional data, which consists of a 3D rotation vector and a 3D translation vector to represent the movement of the camera between the two frames; the deep features generated by the training images after the encoder-decoder are fused with the features generated by the pose estimation model to construct a loss function optimization network model, where I1 can be expressed as:

[0078] I1=F(I S ,P,D P ,K)

[0079] The minimum photometric error loss function in the image reconstruction loss in this method is:

[0080] L s (I1,I2)=L s (F(I S ,P,D P ,K),I2)

[0081] Where I1 is the reconstructed target image, I2 is the target image, P is the pose estimation, and D P is the predicted depth of the target image, and K is the internal parameter of the camera.

[0082] PoseNet combines the internal parameters of the camera (also called the intrinsic parameter matrix). Generally, the form of the intrinsic parameter matrix is ​​as follows:

[0083]

[0084] Where: f x and f y is the focal length of the camera in the x and y directions (in pixels), c x and c y are the coordinates of the optical center of the image (usually at the center of the image). These parameters can be obtained through camera calibration, especially when calculating 3D pose estimation, object tracking or other applications that require accurate spatial position information.

[0085] L s(I1, I2) is calculated by weighted summation using structural similarity and the L1 norm between I1 and I2, which can be expressed as:

[0086]

[0087] The total loss of the model can be expressed as:

[0088]

[0089] Where L smooth It can be expressed as:

[0090]

[0091] in, represents the normalized inverse depth map, and They represent the steps of finding partial derivatives for the width and height of the image respectively, e is the natural exponent, and I2 is the target image.

[0092] See attached Figure 5 , Step S3: Use the public KITTI dataset to train the constructed model to obtain a weight model, and evaluate the accuracy of the trained model for monocular image depth estimation.

[0093] The model training uses the public KITTI dataset to train the constructed model to obtain the weight model. The training dataset is 4424 images and the test dataset is 697 images.

[0094] The present invention trains the model on NVIDIA A100, sets the batch size (Batch Size) of training learning to 32, the training cycle (Epoch) to 60, and the initialization learning rate to 0.00005.

[0095] Since the monocular depth estimation method based on the bidirectional state space model can effectively model long sequences of images, and the method does not generate secondary computational complexity like the attention mechanism, it enhances the model's ability to estimate the depth of monocular images and extract fine-grained local features more finely, while also reducing the difficulty of model training and improving the model's reasoning speed, thus solving the problem of poor performance of monocular depth estimation of images. Compared with the existing technology, the accuracy of monocular depth estimation of images is improved.

[0096] An embodiment of the present application discloses a monocular depth estimation device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the monocular depth estimation method based on the bidirectional state space model as described above.

[0097] An embodiment of the present application discloses a storage medium, which is a computer-readable storage medium. A computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the image classification method based on the deep learning model as described above are implemented.

[0098] Although the present invention is described herein with reference to specific embodiments, it should be understood that these embodiments are merely examples of the principles and applications of the present invention. It should therefore be understood that many modifications may be made to the exemplary embodiments and that other arrangements may be devised without departing from the spirit and scope of the present invention as defined by the appended claims. It should be understood that the various dependent claims and features described herein may be combined in a manner different from that described in the original claims. It should also be understood that the features described in conjunction with a single embodiment may be used in other described embodiments.

Claims

1. A monocular depth estimation method based on a bidirectional state space model, characterized in that: The steps include: Step S1: construct an encoder-decoder structure, including a continuous dilated convolution module and a bidirectional state space module, the encoder generates intermediate features of the training image, models the long-distance intermediate feature information, and the decoder generates output features; Step S2: construct a pose estimation model, and use the output features generated by the decoder and the depth estimation of the training image to jointly reconstruct the learning target, design the loss function of the target image and the reconstructed image to optimize the network; Step S3: Use the public KITTI dataset to train the constructed model to obtain a weight model, and evaluate the accuracy of the trained model for monocular image depth estimation.

2. The monocular depth estimation method based on a bidirectional state space model according to claim 1, characterized in that: The step S1 constructs an encoder-decoder structure, wherein the encoder generates intermediate features of the training image, models the long-distance intermediate feature information, and the decoder generates output features, including: The backbone of the encoder is constructed, wherein the backbone of the encoder includes a convolution module (Conv Stem) and an intermediate feature extraction layer, wherein the intermediate extraction layer includes a continuously dilated convolution module and a bidirectional state space module; after a training image is processed by the convolution module (Conv Stem), a feature matrix that can be processed by the bidirectional state space module is generated by the continuously dilated convolution module; the output feature matrix is ​​input to the bidirectional state space module for processing to model long-distance intermediate feature information; based on the feature matrix output by the encoder, depth information of the training image is generated by decoding through a three-layer decoder.

3. The monocular depth estimation method based on a bidirectional state space model according to claim 2, characterized in that: The step of generating a feature matrix that can be processed by the bidirectional state space module through the continuous dilation convolution module after the training image is processed by the convolution module (Conv Stem) includes: The image processed by the convolution module (Conv Stem) is extracted through depth-wise separable dilated convolution to generate a feature matrix with multi-scale local information; the batch normalization layer (BatchNorm) vertically normalizes the multi-scale local features to obtain normalized features; the normalized features generate a new feature matrix (FeatureMap) through a one-dimensional convolution kernel; the new feature matrix is ​​activated by a RELU activation function; the features activated by the RELU activation function are passed through a one-dimensional convolution kernel to generate a feature matrix that can be processed by a bidirectional state space module.

4. The monocular depth estimation method based on a bidirectional state space model according to claim 3, characterized in that: The output feature matrix is ​​input to the bidirectional state space module for processing, and the step of modeling the long-distance intermediate feature information includes: receiving a feature matrix with a dimension of [B, C, H, W] generated by a continuous expansion convolution module, wherein B is the batch size, C is the number of channels, H is the height of the feature map, and W is the width of the feature map; processing the feature map through the convolution module to obtain a feature matrix output by the convolution layer; the feature matrix output by the convolution layer is respectively generated through the fully connected layer 1 and the fully connected layer 2 to generate two feature branches x1 and x2, applying the ReLU6 activation function to the branch x1, and multiplying it with x2 to obtain a fused feature matrix with a shape of [B, C, H, W]; The fused feature matrix is ​​further processed by deep convolution without batch normalization to obtain a new feature with a shape of [B, C, H, W]. The new feature map with a shape of [B, C, H, W] is transformed in dimension, and the matrix shape changes from [B, C, H, W] to [B, H*W, C] to obtain a feature matrix with a shape of [B, H*W, C]. The feature matrix with a shape of [B, H*W, C] is input into the projection layer of the bidirectional state space module, and the received input features are linearly projected, and the feature dimension is expanded to twice the original dimension for subsequent feature processing. The features output by the projection layer are received, and a one-dimensional convolution kernel is applied to perform a convolution operation to capture local spatiotemporal dependencies. The feature sequence after one-dimensional convolution is activated by the SiLU activation function; the activated feature sequence is projected to the state space model, including the time step (dt), the state matrix B and the state matrix C, which are used to update various parameters in the state space, including the bidirectional state space matrix, the convolution block and the fully connected layer, which are used to process the input features from both the forward and reverse directions, model the long-distance intermediate feature information, and obtain the bidirectional state space model output; the input features are directly added to the bidirectional state space model output to form a residual connection to ensure stable gradient propagation; the output features of the bidirectional state space model are reprojected to the input dimension, and the output feature matrix shape is [B, C, H, W].

5. The monocular depth estimation method based on a bidirectional state space model according to claim 4, characterized in that: The batch normalization layer (BatchNorm) vertically normalizes the multi-scale local features to obtain normalized features. The specific steps are as follows: a) Save the input X for residual connection operation; b)DDWConv r (·) is the dilated convolutional layer, which can expand the receptive field without increasing the number of parameters. The output dimension remains unchanged, which is still [B, C, H, W]; c) The batch normalization layer (BatchNorm) vertically normalizes the multi-scale local features to obtain normalized features, which are expressed as: in, is the sample mean of the mini-batch sample B, is the sample standard deviation of sample B. After applying normalization, the mean of the generated mini-batch is 0 and the unit variance is 1. The scaling parameter γ and the offset parameter β are defined, and the shape is the same as the input parameter X; d)Linear G1 For the point-by-point convolution layer 1, the feature dimension of each pixel position is expanded from C to 6C, which can improve the feature representation capability; e) ReLU is the activation function, expressed as ReLU(x)=max(0,x); f)Linear G2 For the point-by-point convolution layer 2, the feature dimension of each pixel is restored from 6C to C, and the initial number of channels is restored; g) Perform residual connection, add the input X to the regularized output to form a skip connection, and finally get the output of the continuous expansion convolution module The shapes are [B,C,H,W] and [B,C,H in ,W in ]Stay consistent.

6. The monocular depth estimation method based on a bidirectional state space model according to claim 5, characterized in that: The steps of constructing a posture estimation model in step S2, and jointly reconstructing the depth estimation of the training image using the output features generated by the decoder, modeling the learning target, and designing the loss function of the target image and the reconstructed image to optimize the network include: A pose estimation model (PoseNet) is constructed. The model inputs two adjacent frames of images in the training data set and outputs the relative pose between the two frames. The network extracts high-dimensional features from the input RGB images; the high-dimensional features generate feature maps through the feature extraction layer, and the feature maps are processed through convolution operations to extract information about the camera motion in the image; the model finally outputs a pose vector containing six-dimensional data, which consists of a 3D rotation vector and a 3D translation vector, and is used to represent the movement of the camera between the two frames; the deep features generated by the training images through the encoder-decoder are fused with the features generated by the pose estimation model to construct a loss function optimization network model.

7. The monocular depth estimation method based on a bidirectional state space model according to claim 6, characterized in that: The step of fusing the deep features generated by the training image through the encoder-decoder with the features generated by the posture estimation model to construct a loss function optimization network model comprises: I1=F(I S ,P,D P ,K) The minimum photometric error loss function in image reconstruction loss is: L s (I1,I2)=L s (F(I S ,P,D P ,K),I2) Where I1 is the reconstructed target image which can be calculated by the F function, I2 is the target image, I S represents the original image of the training input, P is the pose estimation, and D P is the predicted depth of the target image, K is the internal parameter of the camera, α is the hyperparameter set to 0.85; L s (I1, I2) is calculated by weighted summation using structural similarity and the L1 norm between I1 and I2, expressed as: The total loss of the model is expressed as: Where L smooth It is expressed as: in, represents the normalized inverse depth map, and They represent the steps of finding partial derivatives for the width and height of the image respectively, e is the natural exponent, and I2 is the target image.

8. The monocular depth estimation method based on a bidirectional state space model according to claim 7, characterized in that: The high-dimensional features are obtained as follows: In a single-layer neural network, the new operation is expressed as The features of two linear transformations are fused by element-wise multiplication, let And the new operation can be simplified to definition C is the number of channels of the input feature, the superscript T indicates the transposition of the matrix, i and j are used to index the corresponding channel number, and the star operation is expressed as the following equation: in: Extended to a 3-layer network structure, as follows: Accepts features of dimension [B, C, H, W] generated by continuous dilated convolution modules The features are processed through the convolution module to obtain the feature matrix output by the convolution layer; The feature matrix output by the convolutional layer generates two feature branches x1 and x2 respectively through the fully connected layer 1 and the fully connected layer 2. The ReLU6 activation function is applied to branch x1 and multiplied with x2 to obtain a fused feature matrix with a shape of [B, C, H, W]. The fused feature matrix is ​​further processed by deep convolution without batch normalization to obtain a new feature with the shape of [B, C, H, W]. Will The dimensions are resized to [B, C, H*W], the height and width are flattened and merged into one dimension H*W, and then the feature tensor dimensions are rearranged, [B, C, H*W] is rearranged to [B, H*W, C], and the new The shape is [B, H*W, C], which is convenient for passing to the selected state space model for processing; After adjusting the dimensions Projecting to the fully connected layer obtains features B, C and △, expressed as: The state space model parameter matrix is ​​obtained by calculating the characteristics B, C and △ It is expressed as: For the state-space model parameter matrix And feature U to model the state space model, output That is, the input feature The output features after long sequence modeling by the state space model (SSM) are The calculation process is expressed as:

9. A monocular depth estimation device, characterized in that: It comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the monocular depth estimation method based on the bidirectional state space model as described in any one of claims 1 to 8.

10. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by the processor, the steps of the monocular depth estimation method based on the bidirectional state space model described in any one of claims 1 to 8 are implemented.

Images