Self-supervised monocular depth estimation method and system based on state space model

Abstract

The invention discloses a self-supervised monocular depth estimation method and system based on a state space model, and belongs to the field of computer vision and deep learning. The method comprises the following steps: performing down-sampling operation on an image sequence by using an initial network to obtain a three-scale feature sequence; respectively performing down-sampling operation on the three-scale feature sequences output by the initial network by using a hidden state layer, and then performing block processing operation to obtain a multi-scale feature map; inputting adjacent target image frames into the attitude network to obtain a six-degree-of-freedom attitude transformation matrix; obtaining a depth map through the multi-scale feature map and the depth network, and finally obtaining a final result through the internal reference network. According to the method, the pose network, the depth network and the internal reference network are integrated together, and compared with an existing lightweight monocular depth estimation method, the method comprehensively surpasses an existing model under the same parameter magnitude; according to the method, under the same or smaller parameter quantity, the overall error is lower, and the precision is higher.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision and deep learning, and in particular to a self-supervised monocular depth estimation method and system based on a state-space model. Background Art

[0002] Depth estimation is crucial for many applications, such as autonomous driving, SLAM, and 3D scene reconstruction. To deploy algorithms on industrial edge devices, depth estimation is often performed using standard images. Compared to stereo images, monocular video data is easier to collect and does not require rigorous multi-camera parameter calibration. Therefore, self-supervised monocular depth estimation strategies based on adjacent video frames have become mainstream in this field.

[0003] However, in existing self-supervised MDE methods, the depth estimation network and the pose estimation network are typically separated, and this separate architecture does not meet practical requirements. Depth and pose are closely related, and the lack of a direct connection leads to insufficient information flow between the networks, which affects the accuracy and generalization of the final depth estimation. Existing self-supervised methods typically rely on known camera intrinsic parameters to calculate the image reconstruction loss. However, due to the differences in intrinsic parameters of different cameras, the measurement ambiguity problem makes these methods unable to effectively adapt to the parameters of different cameras during training, resulting in poor generalization ability on different datasets, especially when the camera intrinsic parameters are not known. In other words, the model lacks adaptability and it is difficult to perform effective inference across datasets. Summary of the Invention

[0004] To solve the above technical problems, the present invention provides a self-supervised monocular depth estimation method and system based on a state-space model.

[0005] To implement the above technology, the details are as follows:

[0006] A self-supervised monocular depth estimation method based on a state-space model comprises the following steps:

[0007] S1. Use the initial network to downsample the image sequence to obtain feature sequences at three scales;

[0008] The image sequence is a continuous sequence of n frames, represented by {I1,I2,…,I n}, where each frame image I∈R H×W×C , H and W represent the height and width of the image respectively, and C is the number of channels;

[0009] The initial network performs downsampling operations on the image sequence at three scales;

[0010] The downsampling operation is as follows: downsample the input image sequence by 2 times, and output the first feature sequence with a scale of H / 2 and W / 2; downsample the first scale feature sequence by 2 times to obtain the second feature sequence with a scale of H / 4 and W / 4; downsample the second scale feature sequence by 2 times to obtain the second feature sequence with a scale of H / 8 and W / 8;

[0011] The feature sequence obtained by the three-scale downsampling operation is used as the input feature of the hidden state layer;

[0012] The images with different downsampling rates representing the same content at the same moment in the feature sequence obtained by the three-scale downsampling operation are set as the target frame, and the features of the three scales of the target frame are used as the input features of the deep network.

[0013] S2. Use the hidden state layer to perform downsampling operations on the feature sequences of the three scales output by the initial network at the same downsampling rate, and then perform block processing operations. The results obtained after the block processing operation are input into the gated recurrent hold mechanism (SGR) module to extract the temporal cumulative posture features to obtain a multi-scale feature map; wherein, the results obtained by the gated recurrent hold mechanism (SGR) module of the previous layer after the downsampling operation are inter-level fused with the results obtained by the block processing operation of the next layer;

[0014] The division method is as follows: each frame image in the three scale feature sequence graphs output by the initial network is divided into patches of the same size;

[0015] The extraction method is as follows: the outermost patch blocks of each image are selected from the divided feature sequence graph in an order from the outside to the inside, and they are combined into a set of inputs, which are then input into the gated recurrent hold mechanism (SGR) to predict the cumulative bit pose features of the image sequence;

[0016] The hidden state layer consists of three stacked gated recurrent retention (SGR) modules. Cross-layer connections are used between layers to achieve inter-level fusion of the downsampling of the previous layer output and the block features of the next layer.

[0017] The gated loop hold mechanism (SGR) module is used to simulate the state equation calculation process in state space models (SSMs). The expression is as follows:

[0018] h k =Ah k-1 +Bx k

[0019] y k =Ch k +Dx k

[0020] Where hk is the accumulated pose feature at time k, h k-1 is the accumulated pose feature at time k-1; x k is the input feature at the current moment; A is the hidden state matrix, which obtains the accumulated pose information through the hidden state layer; B is the input mapping matrix, which is used to represent the features of the current frame; C is the observation matrix, which is used to represent the influence of historical information on the prediction results; D is the feedforward matrix, which is used to represent the depth information of the current feature; y k is the output depth feature;

[0021] In order to enhance the feature fusion capability, the hidden state layer fuses the output of each layer with the input of the next layer through cross-layer connections. The expression is as follows:

[0022]

[0023] Where l represents the current number of layers, which has three layers; Concat represents the feature cascade operation; Represents the accumulated pose features of layer l at time k; Represents the input features of the l+1th layer;

[0024] The hidden state layer outputs multi-scale feature maps including: Downsampling result graph, Downsampling result graph, Downsampling result image.

[0025] S3, inputting two consecutive adjacent target image frames in the multi-scale accumulated posture features of the multi-scale feature sequence obtained through the hidden state layer into the posture network to obtain a 6-degree-of-freedom (DoF) posture transformation matrix;

[0026] Two adjacent target image frames are represented as F i and F i+1 , where i, i+1∈n, n represents the length of the multi-scale feature sequence graph sequence, which is the same as the number of input image frame sequences;

[0027] The decoder in the pose network contains three convolutional layers, each of which introduces a pixel shuffle layer to upgrade the low-resolution feature map to a higher-resolution feature map.

[0028] The posture transformation matrix is obtained by inputting two consecutive adjacent target image frames in each feature sequence graph in the hidden state layer into the three convolutional layers in the decoder of the posture network for decoding operation. The three-layer input data are respectively The downsampling result map of the first convolution layer will The downsampling result of the two consecutive adjacent target image frames is spliced to obtain the first layer of spliced feature vectors, and a 3×3 convolution module is used to extract the features to obtain the first layer of features. The extracted features are passed through the pixel reassembly layer and then used to input the second convolution layer; the second convolution layer The downsampling result of the two consecutive adjacent target image frames is spliced to obtain the second layer of spliced feature vectors. After the result of the first convolution layer passes through the pixel reassembly layer and the second layer of spliced feature vectors, a 3×3 convolution module is used to extract the features to obtain the second layer of features. The extracted features are passed through the pixel reassembly layer and used to input the third convolution layer. The third convolution layer will In the downsampling result image, two consecutive adjacent target image frames are subjected to feature splicing operation to obtain the third-layer splicing feature vector. After the result of the second convolution layer passes through the pixel reorganization layer and the third-layer splicing feature vector are subjected to feature splicing operation, a 3×3 convolution module is used to perform feature extraction operation to obtain the third-layer feature and then input it into the pixel reorganization layer; finally, the pixel reorganization layer of the third convolution layer passes through a 3×3 convolution module to output a 6-degree-of-freedom (DoF) posture transformation matrix, including the rotation vector R (indicating the 3 degrees of freedom of the object's rotation) and the translation vector t (indicating the translation of the object in three-dimensional space, with 3 degrees of freedom); the purpose is to generate higher-resolution feature representation through efficient reconstruction of low-resolution features to adapt to the posture estimation task.

[0029] S4. Input the feature sequence obtained by the three-scale downsampling operation of the initial network into the deep network, and perform upsampling operation through the deep upsampling module in the deep network; perform residual connection operation on the multi-scale accumulated posture features of the multi-scale feature sequence obtained by the hidden state layer; input the results of the deep upsampling module and the residual connection results into the depth prediction head for upsampling operation to obtain a multi-scale depth map;

[0030] Here are the steps:

[0031] S4.1. Use the deep encoder in the deep network to extract the downsampled 1 / 4 resolution feature map and the downsampled 1 / 8 resolution feature map from the multi-scale feature sequence map generated by the initial network; input the extracted 1 / 4 resolution and 1 / 8 resolution feature maps into the deep upsampling module, upsample the 1 / 8 resolution deep feature map to 1 / 4 resolution, and then splice it with the input 1 / 4 resolution feature map in the channel dimension to obtain a 1 / 4 resolution feature map;

[0032] S4.2. Input the 1 / 32 resolution features and 1 / 16 resolution features from the multi-scale feature sequence map after hidden state layer extraction and division into the residual connection module. The 1 / 32 resolution features are restored to 1 / 16 resolution after pixel reorganization and then restored to 1 / 8 resolution with the 1 / 16 feature map from the multi-scale feature sequence map through residual connection. The resulting 1 / 8 resolution state residual feature map is then upsampled and restored to 1 / 8 resolution.

[0033] S4.3. Input the 1 / 4 resolution feature map and the 1 / 8 resolution state residual feature map into the depth prediction head, upsample the 1 / 8 resolution state residual feature map again to restore it to 1 / 4 resolution, extract the depth features through a 3×3 convolutional layer, and finally use sigmoid to normalize the depth features and output a multi-scale depth map with 1 / 2 resolution.

[0034] S5. Perform image synthesis and reconstruction training on the multi-scale feature map obtained from the hidden state layer and the main network to obtain a state feature matrix; predict the camera intrinsic parameter K through the state feature matrix and the intrinsic parameter network, obtain the camera intrinsic parameter matrix through the camera intrinsic parameter K, and optimize the obtained intrinsic parameter matrix;

[0035] The steps for optimizing training are as follows:

[0036] S5.1. Define the main network, including the initial network, deep network, hidden state layer, depth prediction head, and pose network. Perform steps S1 to S4 on the main network and perform image synthesis and reconstruction training with real camera intrinsic parameters to obtain the state feature matrix.

[0037] S5.2. Freeze the parameters obtained from the main network in S5.1, fix the main network parameters, use the obtained state feature matrix as the input of the intrinsic reference network, and obtain the predicted camera intrinsic parameter K. Minimize the predicted camera intrinsic parameter K and the actual camera intrinsic parameter to train the intrinsic reference network separately.

[0038] S5.3. Jointly train the main network and the internal reference network to optimize the internal reference matrix.

[0039] S6. Convert the images of adjacent target image frames into 3D point clouds using the obtained intrinsic parameter matrix and multi-scale depth map. Then convert the adjacent point clouds into the spatial coordinate system of the target frame using the obtained state feature matrix. Finally, perform the inverse operation of the point cloud projection to obtain the final result.

[0040] S7, calculate the reconstruction loss, edge-aware smoothing loss and total loss of the image sequence in S6, and verify the image reconstruction results;

[0041] S7.1. Obtain the image reconstruction loss L by performing minimum sampling on the photometric reprojection loss and applying a binary automatic mask. r, photometric reprojection loss The expression is as follows:

[0042]

[0043] In the formula, α represents the weight coefficient, which is set to 0.85; SSIM represents the structural similarity loss; I t a target image frame representing an image sequence; represents a composite image frame of the S6 image sequence;

[0044] To L p Minimum sampling is performed to process out-of-view pixels and occluded objects in the source image. The formula is further defined as:

[0045]

[0046] Where, “1” and “-1” respectively indicate that the forward and backward adjacent frames are used to generate the reconstructed image; F is represented as the weighted sum between the structural similarity loss (SSIM) and the L1 loss;

[0047] Then use the binary automatic mask μ to remove the moving pixels and get the image reconstruction loss I r :

[0048]

[0049] In the formula, when μ is 0, it means that the synthetic frame With the target frame I t The loss between the two is greater than the target frame I t With adjacent frames (source frames of the synthesized frame I t-1 or I t+1 The loss between

[0050] S7.2. Calculate the edge-aware smoothing loss function L smooth , the expression is:

[0051]

[0052] Where, Used to represent the normalized inverse depth (disparity), d t Indicates the true depth value; Represents the predicted depth value; and Represents the gradient of the depth map in the horizontal and vertical directions;

[0053] S7.3. Calculate the total loss L total , the expression is:

[0054]

[0055] Where sum represents the number of predicted output depth maps of different scales, and λ is the weight of the smoothing loss, which is set to 10. -3 .

[0056] A self-supervised monocular depth estimation system based on a state-space model, comprising: a feature extraction module, a time series accumulation module, a depth detection head module, a posture transformation module, an internal parameter prediction module, and an image reconstruction module;

[0057] The feature extraction module performs the following operations:

[0058] S1, extracting primary features from the input continuous image frame sequence through the initial network to obtain the initial multi-scale feature map sequence;

[0059] The timing accumulation module performs the following operations:

[0060] S2, after inputting the continuous multi-scale feature map sequence into the hidden state layer for image segmentation, the gated recurrent hold mechanism is used to perform cumulative pose prediction on the continuous adjacent feature maps;

[0061] The depth detection module performs the following operations:

[0062] S3. After depth encoding of the multi-scale feature map sequence from the backbone network, it is input into the depth detection head together with the accumulated posture features output by the temporal accumulation module to obtain the depth prediction result;

[0063] The pose transformation module performs the following operations:

[0064] S4. Using the accumulated posture of two consecutive image frames output by the time series accumulation module as the input of the posture transformation module, the relative posture of the two consecutive image frames is estimated in six degrees of freedom to obtain an estimated posture transformation matrix;

[0065] The internal parameter prediction module performs the following operations:

[0066] S5. All accumulated postures output by the time series accumulation module are used as the input of the internal parameter prediction module. A lightweight decoding head is constructed by using depthwise separable convolution. The predicted internal parameters are obtained through the main network independent pre-training stage, the main network parameter freezing stage, and the overall model joint optimization stage.

[0067] The image reconstruction module performs the following operations:

[0068] S6. Convert the results of the internal parameter prediction module and the depth detection module into 3D point cloud form, and then convert the adjacent point clouds into the spatial coordinate system of the target frame through the output of the posture transformation module to obtain the final result.

[0069] Beneficial effects of the present invention:

[0070] The present invention modifies the classic model framework and uses an improved linear attention mechanism to simulate the calculation process of Space State Models (SSMs); the present invention also expands adjacent frames into a unidirectional sequence consisting of multiple continuous frames; the present invention integrates the pose network, the depth network and the intrinsic parameter network, thereby strengthening the feature correlation between depth, pose and intrinsic parameters. This method not only solves the inefficiency of information utilization caused by the complete separation of the pose network and the depth network, but also enables the predicted intrinsic parameter depth to influence each other. Compared with the existing lightweight monocular depth estimation methods Lite-Mono and MonoRetNet, the present invention comprehensively surpasses the existing models at the same parameter level; the method of the present invention has lower overall error and higher accuracy at the same or smaller parameter amount. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] Figure 1 is a flow chart of the steps of the present invention;

[0072] Figure 2 It is a network structure diagram of the present invention;

[0073] Figure 3 Detailed network structure diagram of the hidden state layer of the present invention;

[0074] Figure 4 This is a depth prediction result diagram of the present invention. DETAILED DESCRIPTION

[0075] The present invention uses KITTI as a training data set, and the present invention is further described in detail below in conjunction with specific embodiments.

[0076] Example 1:

[0077] like Figure 1 and Figure 2 As shown, a self-supervised monocular depth estimation method based on a state-space model includes the following steps:

[0078] S1. Use the initial network to downsample the image sequence to obtain feature sequences at three scales;

[0079] The image sequence is a continuous sequence of n frames, represented by {I1,I2,…,I n}, where each frame image I∈R H×W×C , H and W represent the height and width of the image respectively, and C is the number of channels;

[0080] In this embodiment, H is 640 pixels, W is 192 pixels, C is 3, and n is 9;

[0081] The initial network performs downsampling operations on the image sequence at three scales;

[0082] The downsampling operation is as follows: downsample the input image sequence by 2 times, and output the first feature sequence with a scale of H / 2 and W / 2; downsample the first scale feature sequence by 2 times to obtain the second feature sequence with a scale of H / 4 and W / 4; downsample the second scale feature sequence by 2 times to obtain the second feature sequence with a scale of H / 8 and W / 8;

[0083] The feature sequence obtained by the three-scale downsampling operation is used as the input feature of the hidden state layer;

[0084] The images with different downsampling rates representing the same content at the same time in the feature sequence obtained by the three-scale downsampling operation are set as the target frame, and the features of the three scales of the target frame are used as the input features of the deep network;

[0085] S2, such as Figure 3 As shown in the figure, the hidden state layer is used to perform downsampling operations on the feature sequences of the three scales output by the initial network at the same downsampling rate, and then a block processing operation is performed. The results obtained after the block processing operation are input into the gated recurrent holding mechanism (SGR) module to extract the temporal accumulation posture features to obtain a multi-scale feature map; wherein, the gated recurrent holding mechanism (SGR) module of the upper layer after the downsampling operation is inter-stage fused with the results obtained after the block processing operation of the lower layer;

[0086] The division method is as follows: each frame image in the three scale feature sequence graphs output by the initial network is divided into patches of the same size;

[0087] In this embodiment, the size of the patch is 8*8 pixels;

[0088] The extraction method is as follows: the outermost patch blocks of each image are selected from the divided feature sequence graph in an order from the outside to the inside, and they are combined into a set of inputs, which are then input into the gated recurrent hold mechanism (SGR) to predict the cumulative bit pose features of the image sequence;

[0089] The hidden state layer consists of three stacked gated recurrent retention (SGR) modules. Cross-layer connections are used between layers to achieve inter-level fusion of the downsampling of the previous layer output and the block features of the next layer.

[0090] The gated loop hold mechanism (SGR) module is used to simulate the state equation calculation process in state space models (SSMs). The expression is as follows:

[0091] h k =Ah k-1 +Bxk

[0092] y k =Ch k +Dx k

[0093] Where h k is the accumulated pose feature at time k, h k-1 is the accumulated pose feature at time k-1; x k is the input feature at the current moment; A is the hidden state matrix, which obtains the accumulated pose information through the hidden state layer; B is the input mapping matrix, which is used to represent the features of the current frame; C is the observation matrix, which is used to represent the influence of historical information on the prediction results; D is the feedforward matrix, which is used to represent the depth information of the current feature; y k is the output depth feature;

[0094] In order to enhance the feature fusion capability, the hidden state layer fuses the output of each layer with the input of the next layer through cross-layer connections. The expression is as follows:

[0095]

[0096] Where l represents the current number of layers, which has three layers; Concat represents the feature cascade operation; Represents the accumulated pose features of layer l at time k; Represents the input features of the l+1th layer;

[0097] The hidden state layer outputs multi-scale feature maps including: Downsampling result graph, Downsampling result graph, Downsampling result image.

[0098] S3, inputting two consecutive adjacent target image frames in the multi-scale accumulated posture features of the multi-scale feature sequence obtained through the hidden state layer into the posture network to obtain a 6-degree-of-freedom (DoF) posture transformation matrix;

[0099] Two adjacent target image frames are represented as F i and F i+1 , where i, i+1∈n, n represents the length of the multi-scale feature sequence graph sequence, which is the same as the number of input image frame sequences;

[0100] The decoder in the pose network contains three convolutional layers, each of which introduces a pixel shuffle layer to upgrade the low-resolution feature map to a higher-resolution feature map.

[0101] The posture transformation matrix is obtained by inputting two consecutive adjacent target image frames in each feature sequence graph in the hidden state layer into the three convolutional layers in the decoder of the posture network for decoding operation. The three-layer input data are respectively The downsampling result map of the first convolution layer will The downsampling result of the two consecutive adjacent target image frames is spliced to obtain the first layer of spliced feature vectors, and a 3×3 convolution module is used to extract the features to obtain the first layer of features. The extracted features are passed through the pixel reassembly layer and then used to input the second convolution layer; the second convolution layer The downsampling result of the two consecutive adjacent target image frames is spliced to obtain the second layer of spliced feature vectors. After the result of the first convolution layer passes through the pixel reassembly layer and the second layer of spliced feature vectors, a 3×3 convolution module is used to extract the features to obtain the second layer of features. The extracted features are passed through the pixel reassembly layer and used to input the third convolution layer. The third convolution layer will In the downsampling result image, two consecutive adjacent target image frames are subjected to feature splicing operation to obtain the third-layer splicing feature vector. After the result of the second convolution layer passes through the pixel reorganization layer and the third-layer splicing feature vector are subjected to feature splicing operation, a 3×3 convolution module is used to perform feature extraction operation to obtain the third-layer feature and then input it into the pixel reorganization layer; finally, the pixel reorganization layer of the third convolution layer passes through a 3×3 convolution module to output a 6-degree-of-freedom (DoF) posture transformation matrix, including the rotation vector R (indicating the 3 degrees of freedom of the object's rotation) and the translation vector t (indicating the translation of the object in three-dimensional space, with 3 degrees of freedom); the purpose is to generate higher-resolution feature representation through efficient reconstruction of low-resolution features to adapt to the posture estimation task.

[0102] S4. Input the feature sequence obtained by the three-scale downsampling operation of the initial network into the deep network, and perform upsampling operation through the deep upsampling module in the deep network; perform residual connection operation on the multi-scale accumulated posture features of the multi-scale feature sequence obtained by the hidden state layer; input the results of the deep upsampling module and the residual connection results into the depth prediction head for upsampling operation to obtain a multi-scale depth map;

[0103] Here are the steps:

[0104] S4.1. Use the deep encoder in the deep network to extract the downsampled 1 / 4 resolution feature map and the downsampled 1 / 8 resolution feature map from the multi-scale feature sequence map generated by the initial network; input the extracted 1 / 4 resolution and 1 / 8 resolution feature maps into the deep upsampling module, upsample the 1 / 8 resolution deep feature map to 1 / 4 resolution, and then splice it with the input 1 / 4 resolution feature map in the channel dimension to obtain a 1 / 4 resolution feature map;

[0105] S4.2. Input the 1 / 32 resolution features and 1 / 16 resolution features from the multi-scale feature sequence map after hidden state layer extraction and division into the residual connection module. The 1 / 32 resolution features are restored to 1 / 16 resolution after pixel reorganization and then restored to 1 / 8 resolution with the 1 / 16 feature map from the multi-scale feature sequence map through residual connection. The resulting 1 / 8 resolution state residual feature map is then upsampled and restored to 1 / 8 resolution.

[0106] S4.3. Input the 1 / 4 resolution feature map and the 1 / 8 resolution state residual feature map into the depth prediction head, upsample the 1 / 8 resolution state residual feature map again to restore it to 1 / 4 resolution, extract the depth features through a 3×3 convolutional layer, and finally use sigmoid to normalize the depth features and output a multi-scale depth map with 1 / 2 resolution.

[0107] S5. Perform image synthesis and reconstruction training on the multi-scale feature map obtained from the hidden state layer and the main network to obtain a state feature matrix; predict the camera intrinsic parameter K through the state feature matrix and the intrinsic parameter network, obtain the camera intrinsic parameter matrix through the camera intrinsic parameter K, and optimize the obtained intrinsic parameter matrix;

[0108] Different camera intrinsic parameters will cause different pixel projections on the camera imaging plane after the same camera pose change. This means that the state feature also implicitly contains the camera intrinsic parameter information of the current dataset. Therefore, the continuous state feature matrix can be used to predict the camera intrinsic parameters.

[0109] The steps for optimizing training are as follows:

[0110] S5.1. Define the main network, including the initial network, depth network, hidden state layer, depth prediction head, and pose network. Execute steps S1 to S4 on the main network and perform image synthesis and reconstruction training using real camera intrinsic parameters to obtain the state feature matrix. At this time, the intrinsic reference network does not participate in the parameter update of the overall model.

[0111] In this embodiment, the settings of the main network pre-training phase include: the initial learning rate of the deep network is 1e -4 , the initial learning rate of the rest of the main network is set to 5e -4 ; Weight decay in the main network is set to 10-2 The drop-path rate of the deep network was set to 0.2, and that of the rest of the main network was set to 0.1. The number of epochs per training cycle was 45. To avoid falling into a local optimum without a warm-up phase, two iterations were performed. The iterative learning rate decay of the main network was set to 0.6.

[0112] S5.2. Freeze the parameters obtained from the main network in S5.1, fix the main network parameters, use the obtained state feature matrix as the input of the intrinsic reference network, and obtain the predicted camera intrinsic parameter K. Minimize the predicted camera intrinsic parameter K and the actual camera intrinsic parameter to train the intrinsic reference network separately.

[0113] In this embodiment, the initial learning rate of the internal reference network is set to 1e -3 , weight decay is set to 10 -2 , the Drop-path rate is set to 0.1, and only one training cycle with a total of 30 cycles is performed;

[0114] S5.3. Jointly train the main network and the internal reference network to optimize the internal reference matrix.

[0115] The joint training is: substitute K predicted by the internal reference network into the image synthesis and reconstruction training to realize the joint training of the internal reference network and the main network and optimize the internal reference matrix.

[0116] S6. The images of the adjacent target image frames are converted into 3D point clouds by using the obtained intrinsic parameter matrix and multi-scale depth map. Then, the adjacent point clouds are converted into the spatial coordinate system of the target frame by using the obtained state feature matrix. Finally, the inverse operation of the point cloud projection is performed to obtain the final result, as shown in FIG. Figure 4 shown.

[0117] In this embodiment, the initial learning rates of the main network and the internal reference network are both set to 1e -4 , weight decay is set to 10 -2 , the Drop-path rate is uniformly set to 0.2; a total of 30 cycles of training.

[0118] S7, calculate the reconstruction loss, edge-aware smoothing loss and total loss of the image sequence in S6, and verify the image reconstruction results;

[0119] S7.1. Obtain the image reconstruction loss L by performing minimum sampling on the photometric reprojection loss and applying a binary automatic mask. r , photometric reprojection loss The expression is as follows:

[0120]

[0121] In the formula, α represents the weight coefficient, which is set to 0.85; SSIM represents the structural similarity loss; I t a target image frame representing an image sequence; represents a composite image frame of the S6 image sequence;

[0122] To L p Minimum sampling is performed to process out-of-view pixels and occluded objects in the source image. The formula is further defined as:

[0123]

[0124] Where, “1” and “-1” respectively indicate that the forward and backward adjacent frames are used to generate the reconstructed image; F is represented as the weighted sum between the structural similarity loss (SSIM) and the L1 loss;

[0125] Then use the binary automatic mask μ to remove the moving pixels and get the image reconstruction loss L r :

[0126]

[0127] In the formula, when μ is 0, it means that the synthetic frame With the target frame I t The loss between the two is greater than the target frame I t With adjacent frames (source frames of the synthesized frame I t-1 or I t+1 The loss between

[0128] S7.2. Calculate the edge-aware smoothing loss function L smooth , the expression is:

[0129]

[0130] Where, Used to represent the normalized inverse depth (disparity), d t Indicates the true depth value; Represents the predicted depth value; and Represents the gradient of the depth map in the horizontal and vertical directions;

[0131] S7.3. Calculate the total loss L total , the expression is:

[0132]

[0133] Where sum represents the number of predicted output depth maps of different scales, and λ is the weight of the smoothing loss, which is set to 10. -3 .

[0134] A self-supervised monocular depth estimation system based on a state-space model, comprising: a feature extraction module, a time series accumulation module, a depth detection head module, a posture transformation module, an internal parameter prediction module, and an image reconstruction module;

[0135] The feature extraction module performs the following operations:

[0136] S1, extracting primary features from the input continuous image frame sequence through the initial network to obtain the initial multi-scale feature map sequence;

[0137] The timing accumulation module performs the following operations:

[0138] S2, after inputting the continuous multi-scale feature map sequence into the hidden state layer for image segmentation, the gated recurrent hold mechanism is used to perform cumulative pose prediction on the continuous adjacent feature maps;

[0139] The depth detection module performs the following operations:

[0140] S3. After depth encoding of the multi-scale feature map sequence from the backbone network, it is input into the depth detection head together with the accumulated posture features output by the temporal accumulation module to obtain the depth prediction result;

[0141] The pose transformation module performs the following operations:

[0142] S4. Using the accumulated posture of two consecutive image frames output by the time series accumulation module as the input of the posture transformation module, the relative posture of the two consecutive image frames is estimated in six degrees of freedom to obtain an estimated posture transformation matrix;

[0143] The internal parameter prediction module performs the following operations:

[0144] S5. All accumulated postures output by the time series accumulation module are used as the input of the internal parameter prediction module. A lightweight decoding head is constructed by using depthwise separable convolution. The predicted internal parameters are obtained through the main network independent pre-training stage, the main network parameter freezing stage, and the overall model joint optimization stage.

[0145] The image reconstruction module performs the following operations:

[0146] S6. Convert the results of the internal parameter prediction module and the depth detection module into 3D point cloud form, and then convert the adjacent point clouds into the spatial coordinate system of the target frame through the output of the posture transformation module to obtain the final result.

[0147] The present invention is compared with the existing detection models Lite-Mono and MonoRetNet on the KITTI dataset. The results are shown in Table 1 below:

[0148] Table 1: Comparison of experimental results of Example 1

[0149] method AbsRel SqRel RMSE RMSElog <![CDATA[δ1]]> <![CDATA[δ2]]> <![CDATA[δ3]]> Parameter quantity Lite-Mono 0.125 0.935 4.986 0.204 0.853 0.950 0.978 2.2M MonoRetNet 0.124 0.937 4.993 0.204 0.855 0.950 0.978 1.5M The present invention 0.122 0.914 4.921 0.200 0.861 0.951 0.980 2.1M

[0150] In Table 1, the size of all input images is adjusted to 640×192. Among them, the lower the values of absolute relative error (Abs Rel), square relative error (Sq Rel), root mean square error (RMSE), and logarithmic root mean square error (RMSE log), the better. The higher the values of the first accuracy index δ1, the second accuracy index δ2, and the third accuracy index δ3, the better. The first accuracy index δ1, the second accuracy index δ2, and the third accuracy index δ3 represent thresholds of 1.25, 1.25, and 1.25, respectively. 2 and 1.25 3 When , the ratio of the predicted value to the true value is within the range of the sample proportion.

[0151] Example 2:

[0152] This embodiment is similar to embodiment 1, except that different parameters are used to compare the present invention with the existing detection models Lite-Mono and MonoRetNet on the KITTI dataset. The results are shown in Table 2 below:

[0153] Table 2: Comparison of experimental results of Example 2

[0154] method AbsRel SqRel RMSE RMSElog <![CDATA[δ1]]> <![CDATA[δ2]]> <![CDATA[δ3]]> Parameter quantity Lite-Mono 0.123 0.919 4.926 0.202 0.859 0.951 0.977 2.5M MonoRetNet 0.122 0.925 4.959 0.201 0.859 0.951 0.978 2.2M The present invention 0.119 0.862 4.884 0.196 0.864 0.954 0.982 2.5M

[0155] Example 3:

[0156] This embodiment is similar to embodiment 1, except that different parameters are used to compare the present invention with the existing detection models Lite-Mono and MonoRetNet on the KITTI dataset. The results are shown in Table 3 below:

[0157] Table 3: Comparison of experimental results of Example 3

[0158] method AbsRel SqRel RMSE RMSElog <![CDATA[δ1]]> <![CDATA[δ2]]> <![CDATA[δ3]]> Parameter quantity Lite-Mono 0.121 0.876 4.918 0.199 0.859 0.953 0.980 3.1M MonoRetNet 0.121 0.921 4.933 0.198 0.860 0.952 0.980 2.6M The present invention 0.116 0.839 4.786 0.191 0.867 0.956 0.982 3.3M

[0159] In summary, the experimental results show that the model has comprehensively surpassed the baseline model at the same parameter level. The method of the present invention has lower overall error and higher accuracy at the same or smaller parameter levels. Compared with Lite-Mono and MonoRetNet, as an existing lightweight monocular depth estimation method, the present invention has shown strong competitiveness in all scale parameter levels (1.5M~2.2M, 2.2M~2.5M, 2.6~3.3M). In particular, the present invention has achieved the best performance in all indicators when the parameter level is 2.6~3.3M. At the same time, under the condition of lightweight requirements, the present invention is also better than the comparison method of the same level when the parameter level is 2.1M parameters. It shows that the present invention is effective in introducing the idea of spatial state model into the depth estimation task, which significantly improves the generalization ability and depth prediction ability of the model.

Claims

1. A self-supervised monocular depth estimation method based on a state-space model, characterized in that: The following steps are involved: S1. Use the initial network to downsample the image sequence to obtain feature sequences at three scales; S2. Use the hidden state layer to perform downsampling operations on the feature sequences of the three scales output by the initial network at the same downsampling rate, and then perform block processing operations. The results obtained after the block processing operations are input into the gated loop holding mechanism module to extract the temporal accumulation posture features to obtain a multi-scale feature map; S3, inputting two consecutive adjacent target image frames in the multi-scale accumulated posture features of the multi-scale feature sequence obtained through the hidden state layer into the posture network to obtain a 6-degree-of-freedom posture transformation matrix; S4. Input the feature sequence obtained by the three-scale downsampling operation of the initial network into the deep network, and perform upsampling operation through the deep upsampling module in the deep network; perform residual connection operation on the multi-scale accumulated posture features of the multi-scale feature sequence obtained by the hidden state layer; input the results of the deep upsampling module and the residual connection results into the depth prediction head for upsampling operation to obtain a multi-scale depth map; S5. Perform image synthesis and reconstruction training on the multi-scale feature map obtained from the hidden state layer and the main network to obtain a state feature matrix; predict the camera intrinsic parameter K through the state feature matrix and the intrinsic parameter network, obtain the camera intrinsic parameter matrix through the camera intrinsic parameter K, and optimize the obtained intrinsic parameter matrix; S6. Convert the images of the adjacent target image frames into 3D point clouds using the obtained intrinsic parameter matrix and multi-scale depth map, then convert the adjacent point clouds into the spatial coordinate system of the target frame using the obtained state feature matrix, and finally perform an inverse operation of the point cloud projection to obtain the final result. S7. Calculate the reconstruction loss, edge-aware smoothing loss, and total loss of the S6 image sequence to verify the image reconstruction results.

2. According to a self-supervised monocular depth estimation method based on a state-space model, it is characterized by: The initial network is used to downsample the image sequence to obtain feature sequences of three scales. The downsampling operation is as follows: downsampling the input image sequence by 2 times to output a feature sequence of the first scale H / 2 and W / 2; downsampling the first scale feature sequence by 2 times to obtain a feature sequence of the second scale H / 4 and W / 4; downsampling the second scale feature sequence by 2 times to obtain a feature sequence of the second scale H / 8 and W / 8.

3. According to a self-supervised monocular depth estimation method based on a state-space model, it is characterized by: The method uses a hidden state layer to perform downsampling operations on the feature sequences of the three scales output by the initial network at the same downsampling rate and then performs a block processing operation, and inputs the results obtained after the block processing operation into the gated cycle holding mechanism module to perform a temporal accumulation posture feature extraction operation to obtain a multi-scale feature map; wherein, the method of dividing the inter-level fusion of the results obtained by the gated cycle holding mechanism module after the downsampling operation and the block processing operation of the next layer is as follows: each frame image in the three scale feature sequence maps output by the initial network is divided into patches of the same size; The extraction method is as follows: the outermost patches of each image are selected from the divided feature sequence graph in an order from the outside to the inside, and they are combined into a set of inputs, which are then input into the gated recurrent hold mechanism to predict the cumulative bit pose features of the image sequence; The hidden state layer consists of three stacked gated recurrent retention mechanism modules. Cross-layer connections are used between layers to achieve inter-level fusion of the downsampling of the previous layer output and the block features of the next layer. The gated loop hold mechanism module is used to simulate the state equation calculation process in the state space model. The expression is as follows: h k =Ah k-1 +Bx k y k =Ch k +Dx k Where h k is the accumulated pose feature at time k, h k-1 is the accumulated pose feature at time k-1; x k is the input feature at the current moment; A is the hidden state matrix, which obtains the accumulated pose information through the hidden state layer; B is the input mapping matrix, which is used to represent the features of the current frame; C is the observation matrix, which is used to represent the influence of historical information on the prediction results; D is the feedforward matrix, which is used to represent the depth information of the current feature; y k is the output depth feature; In order to enhance the feature fusion capability, the hidden state layer fuses the output of each layer with the input of the next layer through cross-layer connections. The expression is as follows: Where l represents the current number of layers, which has three layers; Concat represents the feature cascade operation; Represents the accumulated pose features of layer l at time k; Represents the input features of the l+1th layer; The hidden state layer outputs multi-scale feature maps including: Downsampling result graph, Downsampling result graph, Downsampling result image.

4. According to a self-supervised monocular depth estimation method based on a state-space model, it is characterized by: The two consecutive adjacent target image frames in the multi-scale accumulated posture features of the multi-scale feature sequence obtained through the hidden state layer are respectively input into the posture network to obtain a 6-degree-of-freedom posture transformation matrix. The posture transformation matrix is obtained specifically as follows: two consecutive adjacent target image frames in each feature sequence diagram in the hidden state layer are input into the three convolutional layers in the decoder of the posture network for decoding operation, and the three-layer input data are respectively The downsampling result graph; The first convolutional layer will In the downsampling result image, two consecutive adjacent target image frames are spliced to obtain the first layer of spliced feature vectors, and a 3×3 convolution module is used to extract features to obtain the first layer of features. The extracted features are passed through the pixel reassembly layer and then used as input to the second convolution layer. The second convolutional layer will The downsampling result of the two consecutive adjacent target image frames is spliced to obtain the second layer of spliced feature vectors. After the result of the first convolution layer passes through the pixel reorganization layer and the second layer of spliced feature vectors, a 3×3 convolution module is used to perform feature extraction to obtain the second layer of features. The extracted features are passed through the pixel reorganization layer and used as input. The third convolutional layer; the third convolutional layer will In the downsampling result figure, two consecutive adjacent target image frames are subjected to feature splicing operation to obtain the third-layer splicing feature vector. After the result of the second convolution layer passes through the pixel reorganization layer and the third-layer splicing feature vector are subjected to feature splicing operation, a 3×3 convolution module is used to perform feature extraction operation to obtain the third-layer feature and then input it into the pixel reorganization layer; finally, the pixel reorganization layer of the third convolution layer passes through a 3×3 convolution module to output a 6-degree-of-freedom posture transformation matrix, including the rotation vector R and the translation vector t.

5. According to a self-supervised monocular depth estimation method based on a state-space model, it is characterized by: The steps of inputting the feature sequence obtained by the three-scale downsampling operation of the initial network into the deep network, performing an upsampling operation through the deep upsampling module in the deep network; performing a residual connection operation on the multi-scale accumulated posture features of the multi-scale feature sequence obtained by the hidden state layer; and inputting the results of the deep upsampling module and the residual connection into the depth prediction head for upsampling operation to obtain a multi-scale depth map are as follows: S4.

1. Use the deep encoder in the deep network to extract the downsampled 1 / 4 resolution feature map and the downsampled 1 / 8 resolution feature map from the multi-scale feature sequence map generated by the initial network; input the extracted 1 / 4 resolution and 1 / 8 resolution feature maps into the deep upsampling module, upsample the 1 / 8 resolution deep feature map to 1 / 4 resolution, and then splice it with the input 1 / 4 resolution feature map in the channel dimension to obtain a 1 / 4 resolution feature map; S4.

2. Input the 1 / 32 resolution features and 1 / 16 resolution features from the multi-scale feature sequence map after hidden state layer extraction and division into the residual connection module. The 1 / 32 resolution features are restored to 1 / 16 resolution after pixel reorganization and then restored to 1 / 8 resolution with the 1 / 16 feature map from the multi-scale feature sequence map through residual connection. The resulting 1 / 8 resolution state residual feature map is then upsampled and restored to 1 / 8 resolution. S4.

3. Input the 1 / 4 resolution feature map and the 1 / 8 resolution state residual feature map into the depth prediction head, upsample the 1 / 8 resolution state residual feature map again to restore it to 1 / 4 resolution, extract the depth features through a 3×3 convolutional layer, and finally use sigmoid to normalize the depth features and output a multi-scale depth map with 1 / 2 resolution.

6. According to a self-supervised monocular depth estimation method based on a state-space model, it is characterized by: The multi-scale feature map obtained from the hidden state layer is trained with the main network for image synthesis and reconstruction to obtain a state feature matrix; the camera intrinsic parameter K is predicted by the state feature matrix and the intrinsic parameter network, the camera intrinsic parameter matrix is obtained by the camera intrinsic parameter K, and the obtained intrinsic parameter matrix is optimized and trained as follows: S5.

1. Define the main network, including the initial network, depth network, hidden state layer, depth prediction head, and pose network. Execute steps S1 to S4 on the main network and perform image synthesis and reconstruction training using real camera intrinsic parameters to obtain the state feature matrix. At this time, the intrinsic reference network does not participate in the parameter update of the overall model. S5.

2. Freeze the parameters obtained from the main network in S5.1, fix the main network parameters, use the obtained state feature matrix as the input of the intrinsic reference network, and obtain the predicted camera intrinsic parameter K. Minimize the predicted camera intrinsic parameter K and the actual camera intrinsic parameter to train the intrinsic reference network separately. S5.

3. Jointly train the main network and the internal reference network to optimize the internal reference matrix. The joint training is: substitute K predicted by the internal reference network into the image synthesis and reconstruction training to realize the joint training of the internal reference network and the main network and optimize the internal reference matrix.

7. A self-supervised monocular depth estimation system based on a state-space model, characterized in that: include: Feature extraction module, time series accumulation module, depth detection head module, posture transformation module, internal parameter prediction module, image reconstruction module; The feature extraction module performs the following operations: S1, extracting primary features from the input continuous image frame sequence through the initial network to obtain the initial multi-scale feature map sequence; The timing accumulation module performs the following operations: S2, after inputting the continuous multi-scale feature map sequence into the hidden state layer for image segmentation, the gated recurrent hold mechanism is used to perform cumulative pose prediction on the continuous adjacent feature maps; The depth detection module performs the following operations: S3. After depth encoding of the multi-scale feature map sequence from the backbone network, it is input into the depth detection head together with the accumulated posture features output by the temporal accumulation module to obtain the depth prediction result; The pose transformation module performs the following operations: S4. Using the accumulated posture of two consecutive image frames output by the time series accumulation module as the input of the posture transformation module, the relative posture of the two consecutive image frames is estimated in six degrees of freedom to obtain an estimated posture transformation matrix; The internal parameter prediction module performs the following operations: S5. All accumulated postures output by the time series accumulation module are used as the input of the internal parameter prediction module. A lightweight decoding head is constructed by using depthwise separable convolution. The predicted internal parameters are obtained through the main network independent pre-training stage, the main network parameter freezing stage, and the overall model joint optimization stage. The image reconstruction module performs the following operations: S6. Convert the results of the internal parameter prediction module and the depth detection module into 3D point cloud form, and then convert the adjacent point clouds into the spatial coordinate system of the target frame through the output of the posture transformation module to obtain the final result.

Images