Self-supervised monocular depth estimation method based on Laplacian pyramid

Abstract

The present invention discloses a self-supervised monocular depth estimation method based on Laplacian pyramid, which inputs a continuous image sequence, uses a depth estimation network and a pose estimation network to obtain an estimated coarse depth map and pose conversion information, uses the coarse depth map and the pose conversion information to reconstruct the original image, and finally calculates and minimizes the image reconstruction loss. The present invention can calculate a high-precision depth map and has a wide range of application scenarios, including three-dimensional reconstruction, multi-view stereo, digital protection of cultural heritage, metaverse, virtual reality and augmented reality, etc.

Description

Technical Field

[0001] The present invention relates to the fields of depth estimation and three-dimensional reconstruction, and in particular to a self-supervised monocular depth estimation method. Background Art

[0002] The purpose of monocular depth estimation is to predict the depth value of a scene from a single image. It has a wide range of applications, such as autonomous driving, robot navigation, and 3D reconstruction. At present, the mainstream depth estimation methods are mainly divided into: (1) depth estimation methods based on full supervision (2) depth estimation methods based on self-supervision (3) depth estimation methods based on stereo matching. The first two are depth estimation methods based on deep learning, while the latter is a depth estimation method based on stereo geometry. Compared with deep learning, the use of stereo geometry to estimate depth values ​​does not require repeated training, making it more efficient and faster, but the depth map obtained is less accurate. With the rapid development of deep learning technology, research on predicting depth maps from images using convolutional neural networks (CNN) has emerged. That is, through the actual depth values ​​that have been obtained, a convolutional neural network can be trained to predict the depth value of each pixel on the image.

[0003] Since it is very difficult to obtain large-scale accurate ground truth values ​​as supervisory information, an effective self-supervised monocular depth estimation method is urgently needed. The goal of self-supervised monocular depth estimation is to predict the corresponding depth value from a single color image. Since there are no large numbers of real depth values ​​for training, self-supervision requires training two networks, one predicting the depth value and the other predicting the position transformation information between image pairs. Self-supervised learning methods model the depth estimation task as a view synthesis problem and train a neural network model that can predict the depth map by minimizing the image reconstruction loss. Compared with stereo image frame pairs, the method of using continuous monocular images as supervisory information is more attractive, but it also brings new challenges. For example, in addition to training a network to predict depth values, it also needs to train a pose estimation network.

[0004] In recent years, self-supervised monocular depth estimation methods have made good research progress. The following research papers have been published: "MonoViT: Self-Supervised Monocular Depth Estimation with a Vision Transformer" discloses a new monocular depth estimation framework, which combines the global feature interaction that the VIT model is good at with self-supervised monocular depth estimation, and realizes local and global feature interaction by fusing ordinary convolution modules and Transformer modules, thereby achieving a higher level of detail and precision depth prediction. "Channel-WiseAttention-Based Network for Self-Supervised Monocular Depth Estimation" discloses a method of using the self-attention mechanism to capture long-range dependencies, and aggregates feature information on the channel dimension to enhance the perception of scene structure. In addition, in order to highlight key local feature information, the channel feature map is recalibrated and the feature information is selectively emphasized to achieve the effect of fusing features at different levels. "Unsupervised Monocular Depth Estimation with Left-Right Consistency" discloses a new training method that treats the depth estimation problem as a new view synthesis problem. It is trained by minimizing the photometric loss between the original image and the reconstructed image, thereby obtaining a network model that can predict depth maps.

[0005] The above-mentioned monocular depth estimation methods based on deep learning all use convolutional neural networks to extract feature information of the input image in the encoder stage, then obtain a coarse depth map through upsampling operations in the decoder, and reconstruct the image in combination with the pose information obtained by the pose network, and finally train the depth estimation network model by minimizing the loss between the reconstructed image and the original image. However, repeated downsampling operations will lose some feature information, and it is difficult to obtain an accurate depth map using upsampling operations. Summary of the invention

[0006] Purpose of the invention: The purpose of the present invention is to address the deficiencies in self-supervised monocular depth estimation, and to provide a self-supervised monocular depth estimation method based on the Laplacian pyramid, in which the Laplacian residual map is added to different stages of the encoder to enrich the coding features and make the predicted depth map more accurate.

[0007] Technical solution: A self-supervised monocular depth estimation method based on Laplacian pyramid of the present invention comprises the following steps:

[0008] Step S1, inputting a continuous image sequence; the image data in the image sequence may be image data collected by a handheld camera or aerial image data collected by an unmanned aerial vehicle device;

[0009] Step S2: Estimating a rough depth map D t and pose transformation information T t→t′ ;

[0010] The method for estimating the coarse depth map is as follows: transform the color image I in step 1 into t Input depth estimation network, for the input image I t The downsampling operation is performed, and the corresponding Laplace residual is added to the encoder to achieve the effect of enriching the encoding features; the obtained feature map is then passed through the upsampling module and the residual module to obtain the coarse depth map D t ;

[0011] The method for obtaining the pose transformation information is as follows: select a pair of continuous color images I in step 1 t and I t+1 , input it into the pose estimation network, first encode and extract features through the pre-trained ResNet network, and then obtain the pose information through a simple convolution layer. The pose information includes the relative poses of the six degrees of freedom of adjacent images;

[0012] Step S3: Reconstruct the original image using the coarse depth map and pose conversion information

[0013]

[0014] Where t′ is t-1 or t+1, I t Refers to the current frame image; It refers to the original image after reconstruction;

[0015] Step S4: Calculate and minimize the image reconstruction loss by minimizing the photometric reprojection loss pixel by pixel. The final loss function Loss includes the image reconstruction loss and smoothing loss

[0016] Among them, S is the scale number, λ is the weight coefficient of the smoothing regularization term;

[0017] Step S5, training a pose estimation network and a depth estimation network using a continuous image sequence;

[0018] The training data used in the present invention can be either image data collected by a handheld camera or aerial image data collected by a drone;

[0019] Step S6: predicting a depth map using the trained depth estimation network;

[0020] Through steps S1 to S5, a depth estimation network with better performance can be trained, and finally a high-precision depth image can be predicted.

[0021] Furthermore, the step 2 estimates the rough depth map D t Specifically, it includes downsampling to obtain coding features, combining Laplace residual to enrich coding features, and the decoder to obtain a coarse depth map D t Three steps, details are as follows:

[0022] The image I of size H×W×3 t Input the depth estimation network, which is divided into 3 layers, and each layer obtains a depth map of different resolutions;

[0023] First, the image I is convolutionally transformed through three 3×3 convolutional layers with a stride of 1. t The features are extracted and then downsampled using a 2×2 maximum pooling layer with a step size of 2 to obtain a size of feature map; since part of the feature information of the input image, especially the boundary information, will be lost during the downsampling process, and the Laplacian residual map contains the boundary information of the image, the feature map is concatenated with the Laplacian residual map lap1 with 3 channels and enters the first layer; in the first layer, the feature map is first downsampled through a 3×3 convolutional layer with a step size of 1 and a 2×2 maximum pooling layer with a step size of 2 to obtain a size of At the same time, in order to improve the receptive field of the convolution layer, the feature map is sent to the hole convolution module. After the hole convolution module, the feature map is subjected to multi-dimensional feature interaction through the channel attention module, thereby improving the feature extraction capability;

[0024] Subsequently, the interacted features are concatenated with the Laplacian residual Lap2 and the feature map after the first layer of pooling downsampling and enter the second layer. In the second layer, the concatenated features are first downsampled through a 3×3 convolutional layer with a step size of 1 and a 2×2 maximum pooling layer with a step size of 2 to obtain feature map; then it will enter the dilated convolution module composed of three 3×3 dilated convolution layers with a step size of 1 and the channel attention module for feature interaction for feature extraction and interaction; finally, the obtained features and the Laplacian residual Lap3 and the feature map after the second layer of pooling downsampling are spliced ​​into the third layer;

[0025] Similarly, in the third layer, the features are downsampled and then enter the dilated convolution module and the channel attention module to obtain a size of The feature map of

[0026] The encoder will lose some image feature information when extracting features by downsampling. To compensate for the lost feature information, the Laplace residual of the corresponding resolution is added before each downsampling. The maximum pooling layer in U-Net is used instead of the general downsampling convolution layer for downsampling operation, which slows down the speed of feature loss and improves the convergence speed of the model. At the same time, the maximum pooling layer can highlight and amplify the feature information, making it easier for the encoder to extract the feature information required for the training model during the feature extraction process. The missing data in the sampling process is filled with data using the bilinear interpolation method to ensure the accuracy of the data.

[0027] After the features are extracted by the encoder, the feature information first enters the residual module. In the residual module, it first passes through a 3×3 convolution layer with a step size of 1, and then enters the branch structure. After passing through the branch structure, it is added with the previous feature information to form a residual network; the residual network can prevent overfitting caused by too many convolution layers, thereby steadily improving the model effect;

[0028] Subsequently, the data will enter the upsampling layer to restore the resolution. The upsampling layer uses bilinear interpolation to expand the spatial dimension. At the same time, in order to improve the accuracy of the padded data, the upsampled data will pass through a 3×3 convolution layer with a step size of 1 and a 1×1 convolution layer with a step size of 1.

[0029] Finally, the data goes through an upsampling operation and an activation function to obtain the final coarse depth map.

[0030] Furthermore, the method to obtain the Laplace residual graph is:

[0031] First, the image is downsampled to a size of and The downsampled image of size The downsampled image is upsampled and the size is and The up-sampled image is then subtracted from the up-sampled image of the corresponding size to obtain the Laplace residuals lap1, lap2 and lap3.

[0032] Furthermore, the branch structure of the residual module includes two 3×3 convolutional layers with a step size of 1, two Elu activation functions and a BatchNorm layer for normalization; the branch structure can classify the features extracted by the encoder to obtain better prediction results, and after branching, it is added to the previous data to form a residual network.

[0033] Furthermore, the pixel-by-pixel minimization of the photometric reprojection loss in step S4 means

[0034] Among them, K is the intrinsic parameter of the camera, [] is the sampling operator,

[0035] The image reconstruction loss The calculation method is as follows:

[0036] First, by calculating the pixel-level similarity SSIM and and I t The error between them gives Loss:

[0037]

[0038] Then use D t , T t→t′ , K projects the depth map D t in I t′ The two-dimensional coordinates in , and then the final reconstructed image is obtained through the sampling operator However, in real scenes, the camera may be stationary and the scene may be moving, which breaks the requirement of self-supervision for moving cameras and stationary scenes. To solve this problem, the present invention uses an automatic mask to filter out the pixels that remain stationary and in the same scene in consecutive frames.

[0039] At this point, the image reconstruction loss is expressed as:

[0040] In order to make the generated inverse depth map smoother and regularize the disparity of texture-free areas, an edge-aware smoothing regularization term is introduced, and the smoothing loss The calculation method is as follows:

[0041]

[0042] in, dt * represents the average normalized inverse depth, expressed by dt * Prevent the estimated depth value from dropping too fast; and then get the final loss function:

[0043] Beneficial effects: The present invention uses the Laplacian residual graph to enrich the coding features to compensate for the feature information lost in the downsampling process; uses the maximum pooling layer to highlight and amplify the feature information in the downsampling process, so that the encoder can more easily extract the feature information required for the training model in the feature extraction process; uses the residual module to solve the overfitting problem and improve the decoder's efficiency in utilizing features. Compared with the prior art, the present invention has the following advantages:

[0044] (1) The present invention adds Laplace residual to the encoder to enrich the encoding features.

[0045] (2) The present invention uses multiple convolutional layers to extract features in the feature extraction stage, and then uses the maximum pooling layer to perform downsampling operations, which not only prevents the feature from being lost too quickly but also highlights the extracted feature information.

[0046] (3) The present invention introduces a residual module into the upsampling module of the decoder, which not only solves the overfitting problem caused by too many convolutional layers but also improves the decoder's efficiency in utilizing features. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 It is the overall flow chart of the present invention;

[0048] Figure 2 A schematic diagram of a depth estimation network of the present invention;

[0049] Figure 3 Schematic diagram of the posture estimation network of the present invention;

[0050] Figure 4 Schematic diagram of the residual module of the present invention;

[0051] Figure 5 is a sample image used in the embodiment;

[0052] Figure 6 is a sample of the depth map predicted in the embodiment;

[0053] Figure 7 is a schematic diagram of a depth image output in an embodiment;

[0054] Figure 8 It is a depth map obtained by the prior art solution in the embodiment;

[0055] Fig. 9 A flowchart of obtaining a coarse depth map in an embodiment;

[0056] Fig.10 Flow chart of calculating the loss function in the embodiment. DETAILED DESCRIPTION

[0057] The technical solution of the present invention is described in detail below, but the protection scope of the present invention is not limited to the listed embodiments.

[0058] Existing deep learning-based monocular depth estimation methods all use convolutional neural networks to extract feature information of the input image at the encoder stage, then obtain a coarse depth map through upsampling operations in the decoder, and reconstruct the image by combining the pose information obtained by the pose network, and finally train the depth estimation network model by minimizing the loss between the reconstructed image and the original image. However, repeated downsampling operations will lose some feature information, and it is difficult to use upsampling operations to obtain an accurate depth map.

[0059] In order to solve the above technical problems, the present invention proposes a self-supervised monocular depth estimation method based on Laplacian pyramid, which trains the depth estimation network and the pose estimation network by inputting a continuous image sequence, predicts a coarse depth map and pose conversion information from the image sequence, and reconstructs the original image with the coarse depth map and pose conversion information and minimizes the loss between the original image and the reconstructed image, and finally trains the depth estimation network through a data set, and predicts the depth map with the trained network. The present invention realizes the most advanced and lightweight monocular depth estimation. By inputting a single image, the corresponding depth map is accurately predicted; at the same time, the residual map corresponding to the Laplacian pyramid is added to the encoding part of the model to enrich the encoding features and improve the model accuracy, and the maximum pooling layer is used for downsampling to slow down feature loss and improve the model convergence speed, and the residual module is added before the decoder upsampling to improve feature utilization efficiency and improve model accuracy.

[0060] like Figures 1 to 4 As shown, the self-supervised monocular depth estimation method based on the Laplacian pyramid of this embodiment includes the following steps:

[0061] Step S1, inputting a continuous image sequence;

[0062] Step S2: Estimating a rough depth map D t and pose transformation information T t→t′ ;

[0063] The method for estimating the coarse depth map is as follows: transform the color image I in step 1 into t Input depth estimation network, for the input image I t The downsampling operation is performed, and the corresponding Laplace residual is added to the encoder to achieve the effect of enriching the encoding features; the obtained feature map is then passed through the upsampling module and the residual module to obtain the coarse depth map D t ;

[0064] The method for obtaining the pose transformation information is as follows: select a pair of continuous color images I in step 1 t and I t+1 , input it into the pose estimation network, first encode and extract features through the pre-trained ResNet network, and then obtain the pose information through a simple convolution layer. The pose information includes the relative poses of the six degrees of freedom of adjacent images;

[0065] Step S3: Reconstruct the original image using the coarse depth map and pose conversion information

[0066]

[0067] Where t′ is t-1 or t+1, L t Refers to the current frame image; It refers to the original image after reconstruction;

[0068] Step S4: Calculate and minimize the image reconstruction loss by minimizing the photometric reprojection loss pixel by pixel. The final loss function Loss includes the image reconstruction loss and smoothing loss

[0069] Among them, S is the scale number, λ is the weight coefficient of the smoothing regularization term;

[0070] Step S5: train a pose estimation network and a depth estimation network using a continuous image sequence; and use the trained depth estimation network to predict a depth map.

[0071] This embodiment Figure 5 The four scene images in the figure are used for self-supervised monocular depth estimation based on the Laplacian pyramid. The corresponding depth maps are shown in Figure 6 Specifically, it includes downsampling to obtain coding features, combining Laplace residual to enrich coding features, and the decoder to obtain a coarse depth map D t Three steps, details as follows Fig. 9 and Fig.10 As shown:

[0072] The image I of size H×W×3 t Input the depth estimation network, which is divided into 3 layers, and each layer obtains a depth map of different resolutions;

[0073] First, the image I is convolutionally transformed through three 3×3 convolutional layers with a stride of 1. t The features are extracted and then downsampled using a 2×2 maximum pooling layer with a step size of 2 to obtain a size of feature map; since part of the feature information of the input image, especially the boundary information, will be lost during the downsampling process, and the Laplacian residual map contains the boundary information of the image, the feature map is concatenated with the Laplacian residual map lap1 with 3 channels and enters the first layer; in the first layer, the feature map is first downsampled through a 3×3 convolutional layer with a step size of 1 and a 2×2 maximum pooling layer with a step size of 2 to obtain a size of At the same time, in order to improve the receptive field of the convolution layer, the feature map is sent to the hole convolution module. After the hole convolution module, the feature map is subjected to multi-dimensional feature interaction through the channel attention module, thereby improving the feature extraction capability;

[0074] Subsequently, the interacted features are concatenated with the Laplacian residual Lap2 and the feature map after the first layer of pooling downsampling and enter the second layer. In the second layer, the features are first downsampled through a 3×3 convolutional layer with a step size of 1 and a 2×2 maximum pooling layer with a step size of 2. feature map; then it will enter the dilated convolution module composed of three 3×3 dilated convolution layers with a step size of 1 and the channel attention module for feature interaction for feature extraction and interaction; finally, the obtained features and the Laplacian residual Lap3 and the feature map after the second layer of pooling downsampling are spliced ​​into the third layer;

[0075] Similarly, in the third layer, the features are downsampled and then enter the dilated convolution module and the channel attention module to finally obtain a size of The feature map of

[0076] The encoder will lose some image feature information when extracting features by downsampling. To compensate for the lost feature information, the Laplace residual of the corresponding resolution is added before each downsampling. The maximum pooling layer in U-Net is used instead of the general downsampling convolution layer for downsampling operation, which slows down the speed of feature loss and improves the convergence speed of the model. At the same time, the maximum pooling layer can highlight and amplify the feature information, making it easier for the encoder to extract the feature information required for the training model during the feature extraction process. The missing data in the sampling process is filled with data using the bilinear interpolation method to ensure the accuracy of the data.

[0077] After extracting features through the encoder, the feature information first enters the residual module. In the residual module, it first passes through a 3×3 convolution layer with a step size of 1, and then enters the branch structure. After passing through the branch structure, it is added with the previous feature information to form a residual network; the residual network can prevent overfitting caused by too many convolution layers, thereby steadily improving the model effect; then, the data will enter the upsampling layer to restore the resolution, and the upsampling layer uses bilinear interpolation to expand the spatial dimension; at the same time, in order to improve the accuracy of the filled data, the upsampled data will pass through 3×3 convolution layers with a step size of 1 and 1×1 convolution layers with a step size of 1; finally, the data will go through the upsampling operation and activation function to obtain the final coarse depth map.

[0078] The schematic diagram of the depth estimation network in this embodiment is as follows Figure 2 As shown, Figure 2 There are three upsampling modules (upsample) from bottom to top on the middle right side. For the first upsampling module at the bottom, the data received at its bottom is the feature information obtained by the third stage of the encoder. For the first upsampling module at the bottom, the data received on its left is the feature information obtained by the second stage of the encoder. The same applies to the other two upsampling modules.

[0079] like Figure 6 As shown, the calculation steps of the loss function in this embodiment are as follows:

[0080] First, calculate the pixel-level similarity SSIM and and It The error between them, i.e. the pixel-by-pixel minimization of the photometric reprojection loss:

[0081] Among them, K is the intrinsic parameter of the camera, [] is the sampling operator,

[0082] By calculating the pixel-level similarity SSIM and and I t The error between them gives Loss:

[0083]

[0084] First use D t , T t→t′ , K projects the depth map D t in I t′ The two-dimensional coordinates in , and then the final reconstructed image is obtained through the sampling operator

[0085] Calculate the mask μ. The automatic masking method filters out the pixels that remain stationary and in the same scene in consecutive frames.

[0086] The mask μ is calculated as follows:

[0087] At this point, the image reconstruction loss is expressed as:

[0088] Then, an edge-aware smoothing regularization term is obtained. In order to make the generated inverse depth map smoother and regularize the disparity of texture-free areas, an edge-aware smoothing regularization term is introduced:

[0089]

[0090] in, Represents the average normalized inverse depth, which is used to prevent the estimated depth value from decreasing too quickly.

[0091] Finally, the final loss function is calculated. The final loss function Loss is composed of the image reconstruction loss and smoothing loss Common composition:

[0092]

[0093] Among them, S is the scale number and λ is the weight coefficient of the smoothing regularization term.

[0094] Example

[0095] In this embodiment, the input image sample is as follows: Figure 5As shown in FIG. 1 , the selected images are all from different street scenes in the Make3D dataset and are all taken by a motion camera. After the images are processed by the model trained by the present invention, the corresponding depth map is output, such as Figure 6 As shown in FIG. 1 , the depth map accurately describes the three-dimensional information of the object in the corresponding image. In particular, in the third and fourth street view images, the depth map accurately describes the edge of the object.

[0096] The experimental results are as follows Figure 7 and Figure 8 As shown in the figure, at the red box mark, the visualization effect of the present invention is better, and the objects are presented more clearly and accurately. However, some objects are lost in the depth map using the RMSFM3 and RMSEM6 methods. Therefore, the present invention realizes the most advanced and lightweight monocular depth estimation. Our model accurately predicts the corresponding depth map by inputting a single RGB image.

[0097] The evaluation results of the present invention and the existing methods on the KITTI dataset are compared, and the experimental results are shown in Table 1.

[0098] Table 1 Comparison of evaluation results of the present invention and the existing method

[0099]

[0100] In summary, the present invention adds Laplace residuals to the encoder to enrich the coding features, uses multiple convolutional layers to perform feature extraction in the feature extraction stage, and then uses the maximum pooling layer to perform downsampling operations, which not only prevents the features from being lost too quickly but also highlights the extracted feature information; the present invention also introduces a residual module in the upsampling module of the decoder, which not only solves the overfitting problem caused by too many convolutional layers but also improves the decoder's utilization efficiency of features, so as to better utilize the coding feature regression to obtain a more accurate depth map.

Claims

1. A self-supervised monocular depth estimation method based on Laplacian pyramid, characterized in that: The following steps are involved: Step S1, inputting a continuous image sequence; Step S2: Estimation to obtain a rough depth map D t and pose transformation information T t→t′ ; The method for estimating the rough depth map is as follows: transform the color image I in step 1 into t Input depth estimation network, for the input image I t The downsampling operation is performed, and the corresponding Laplace residual is added to the encoder to achieve the effect of enriching the encoding features; the obtained feature map is then passed through the upsampling module and the residual module to obtain the coarse depth map D t ; The method for obtaining the pose transformation information is as follows: select a pair of continuous color images I in step 1 t and I t+1 , input it into the pose estimation network, first encode and extract features through the pre-trained ResNet network, and then obtain the pose information through the convolution layer. The pose information includes the relative poses of the six degrees of freedom of adjacent images; Step S3: Reconstruct the original image using the coarse depth map and pose conversion information Among them, t ′ is t-1 or t+1, I t Refers to the current frame image; It refers to the original image after reconstruction; Step S4: Calculate and minimize the image reconstruction loss by minimizing the photometric reprojection loss pixel by pixel. The final loss function Loss includes the image reconstruction loss and smoothing loss Among them, S is the scale number, λ is the weight coefficient of the smoothing regularization term; Step S5: training a pose estimation network and a depth estimation network using a continuous image sequence; and predicting a depth map using the trained depth estimation network; The step 2 estimates the rough depth map D t The specific steps include downsampling to obtain coding features, combining Laplace residual to enrich coding features, and the decoder to obtain a coarse depth map D t Three steps, details are as follows: The image I of size H×W×3 t Input the depth estimation network, which is divided into 3 layers, and each layer obtains a depth map of different resolutions; First, the image I is convolutionally transformed through three 3×3 convolutional layers with a stride of 1. t The features are extracted and then downsampled using a 2×2 maximum pooling layer with a step size of 2 to obtain a size of The feature map is concatenated with the Laplace residual lap1 with 3 channels and enters the first layer; in the first layer, the feature map is first downsampled by a 3×3 convolutional layer with a step size of 1 and a 2×2 maximum pooling layer with a step size of 2 to obtain a size of At the same time, the feature map is sent to the dilated convolution module. After the dilated convolution module, the feature map undergoes multi-dimensional feature interaction through the channel attention module. Subsequently, the interacted features are concatenated with the Laplacian residual Lap2 and the feature map after the first layer of pooling downsampling and enter the second layer. In the second layer, the concatenated features are first downsampled through a 3×3 convolutional layer with a step size of 1 and a 2×2 maximum pooling layer with a step size of 2 to obtain feature map; then it will enter the dilated convolution module composed of three 3×3 dilated convolution layers with a step size of 1 and the channel attention module for feature interaction for feature extraction and interaction; finally, the obtained features and the Laplacian residual Lap3 and the feature map after the second layer of pooling downsampling are spliced ​​into the third layer; In the third layer, the features are downsampled and enter the hole convolution module and the channel attention module to finally obtain a size of The feature map of Next, the encoder is used to obtain The feature map of is estimated by the decoder to obtain a rough depth map. The method is as follows: the feature information first enters the residual module. In the residual module, it first passes through a 3×3 convolution layer with a step size of 1, and then enters the branch structure. After passing through the branch structure, it is added with the previous feature information to form a residual network; then, the feature information will enter the upsampling layer and the 3×3 convolution layer with a step size of 1 and the 1×1 convolution layer with a step size of 1 in sequence; Finally, the data goes through an upsampling operation and an activation function to obtain the final coarse depth map.

2. The self-supervised monocular depth estimation method based on Laplacian pyramid according to claim 1, characterized in that: The method to obtain the Laplace residual plot is: First, by continuously downsampling the image, the size of and The downsampled image of size The downsampled image is upsampled and the size is and The up-sampled image is then subtracted from the up-sampled image of the corresponding size to obtain the Laplace residuals lap1, lap2 and lap3.

3. The self-supervised monocular depth estimation method based on Laplacian pyramid according to claim 1, characterized in that: The branch structure of the residual module includes two 3×3 convolutional layers with a step size of 1, two Elu activation functions and a BatchNorm layer for normalization; the branch structure can classify the features extracted by the encoder to obtain better prediction results, and after branching, it is added to the previous data to form a residual network.

4. The self-supervised monocular depth estimation method based on Laplacian pyramid according to claim 1, characterized in that: The step S4 minimizes the photometric reprojection loss pixel by pixel, that is, Among them, K is the intrinsic parameter of the camera, [] is the sampling operator, The image reconstruction loss The calculation method is as follows: First, by calculating the pixel-level similarity SSIM and and I t The error between them gives Loss: Then use D t , T t→t′ , K projects the depth map D t in I t ′, and then the final reconstructed image is obtained through the sampling operator Then use the automatic mask to filter the pixels that remain stationary and in the same scene in consecutive frames. At this point, the image reconstruction loss is expressed as: The smoothing loss The calculation method is as follows: in, dt * represents the mean normalized inverse depth; Then we get the final loss function:

Images