Lightweight monocular depth estimation method and device based on self-supervised deep learning

Abstract

The invention relates to a lightweight monocular depth estimation method and device based on self-supervised deep learning, and the method comprises the steps: obtaining a plurality of continuous frames of images, carrying out the reconstruction of a target frame through a trained depth estimation model based on the adjacent frames of the target frame, and obtaining the depth information of the target frame, the training process of the depth estimation model comprises the following steps: obtaining a monocular video sequence, and constructing training data by extracting continuous multi-frame images; predicting relative pose information of two adjacent frames of images through pose coding and decoding; through multi-scale feature extraction and dynamic up-sampling, obtaining a feature map with local detail features reserved, and obtaining a depth map based on the feature map; and performing frame reconstruction based on the relative pose information and the depth map, and calculating a loss function value on the premise of considering triple loss based on semantic perception so as to realize training of a depth estimation model. The method has the advantages that detail features are fully reserved during up-sampling, and the expression effect on gradient-rich areas is good.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a lightweight monocular depth estimation method and device based on self-supervised deep learning. Background Art

[0002] Depth estimation is a classic problem in machine vision, which is very important for 3D reconstruction of scenes, occlusion and lighting processing in augmented reality. In recent years, with the rapid development of deep learning, monocular depth estimation based on deep learning has been widely studied and achieved good accuracy.

[0003] Monocular depth estimation usually uses image data from a single perspective as input and predicts the depth value corresponding to each pixel in the image in an end-to-end manner. The depth value represents the distance from the image collector to each point in the scene. For monocular depth estimation methods based on supervised deep learning, each RGB image needs to have a corresponding depth label. However, the collection of depth labels usually requires a depth camera or lidar. The former is limited by the range, and the latter is expensive. In addition, the original depth labels collected are usually sparse points that cannot be well matched with the original image. In order to avoid the expensive cost of obtaining true depth data, self-supervised depth estimation algorithms have become a research direction.

[0004] Chinese patent application publication number CN114170286A discloses a monocular depth estimation method based on unsupervised deep learning. First, a depth estimation and pose estimation network framework based on unsupervised deep learning is constructed; then the established neural network is trained; finally, the trained network is tested. This method solves the limitations of supervised learning in practical applications while ensuring accuracy. The above application combines photometric loss, smoothing loss and geometric consistency loss to train the neural network to ensure photometric consistency and scale consistency. However, for depth estimation in gradient-rich areas, the effect of the above training scheme is not ideal.

[0005] In summary, there is currently a lack of a monocular depth estimation solution for gradient-rich areas. Summary of the invention

[0006] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and to provide a lightweight monocular depth estimation method and device based on self-supervised deep learning to solve the problem of unsatisfactory monocular depth estimation effect in gradient-rich areas.

[0007] The purpose of the present invention can be achieved by the following technical solutions:

[0008] In one aspect of the present invention, a lightweight monocular depth estimation method based on self-supervised deep learning is provided, wherein a plurality of continuous frames of images are acquired, and based on adjacent frames of the target frame, the target frame is reconstructed using a trained depth estimation model to obtain depth information of the target frame, wherein the training process of the depth estimation model comprises the following steps:

[0009] Obtain a monocular video sequence and construct training data by extracting multiple consecutive frames of images;

[0010] Based on the training data, relative pose information of two adjacent frames of images is predicted through pose encoding and decoding;

[0011] Based on the training data, a feature map retaining local detail features is obtained through multi-scale feature extraction and dynamic upsampling, and a depth map is obtained based on the feature map;

[0012] Frame reconstruction is performed based on the relative pose information and the depth map, and a loss function value is calculated under the premise of considering a semantically-aware triplet loss to implement training of the depth estimation model.

[0013] As a preferred technical solution, the process of obtaining a feature map retaining local detail features based on the training data through multi-scale feature extraction and dynamic upsampling, and obtaining a depth map based on the feature map includes the following steps:

[0014] Based on the training data, obtaining features of multiple scales through deep coding;

[0015] Calculating a bias constant based on the features of the multiple different scales, calculating a sampling group based on the bias constant, and obtaining a feature map retaining local detail features after dynamic upsampling through bilinear interpolation;

[0016] Based on the feature map, a depth map is obtained by generating a disparity map.

[0017] As a preferred technical solution, the dynamic upsampling is achieved by the following formula:

[0018]

[0019]

[0020]

[0021] in, The feature map of size C×H×W after multi-scale feature extraction of the input is The final feature map of size C×sH×sW obtained by dynamic upsampling, s is the sampling ratio, Linear a With Linearb For calculating the bias constant The linear layer has input and output dimensions of C and 2s2 respectively. is the original sampling coordinate, For calculation and The sample set obtained by summing According to the given coordinate grid For the input feature map The function to perform sampling.

[0022] As a preferred technical solution, the calculation process of the semantically-aware triplet loss includes:

[0023] Based on the reconstructed image of the frame, the image is segmented into anchor points, positive samples and negative samples through semantic segmentation;

[0024] By calculating the anchor-positive sample distance and the anchor-negative sample distance, the triplet loss is obtained.

[0025] As a preferred technical solution, the semantically-aware triplet loss is obtained using the following formula:

[0026]

[0027]

[0028]

[0029] in, Indicates the anchor point, is a point with the same semantic information as the anchor point, i.e., a positive sample. are other points with different semantic information from the anchor point, i.e. negative samples. and Represent the distance between anchor point and positive sample and the distance between anchor point and negative sample, respectively. d represents the corresponding deep features, is the set of semantic boundary pixels, m ′ is the threshold, which is used to adjust the minimum interval between the two to prevent a large gap.

[0030] As a preferred technical solution, the loss function value also includes photometric reprojection loss and edge smoothing loss.

[0031] Another aspect of the present invention provides a lightweight monocular depth estimation device based on self-supervised deep learning, comprising:

[0032] The pose estimation module is used to predict the relative pose information of two adjacent frames of images through pose encoding and decoding based on the input of multiple consecutive frames of images;

[0033] A depth estimation module, configured to obtain a feature map retaining local detail features based on the continuous multi-frame images through multi-scale feature extraction and dynamic upsampling, and obtain a depth map based on the feature map;

[0034] A training module is used to perform frame reconstruction based on the relative pose information and the depth map during the training phase, calculate the loss function value under the premise of considering the triplet loss based on semantic perception, and realize the training of the pose estimation module and the depth estimation module.

[0035] As a preferred technical solution, the depth estimation module includes:

[0036] A deep encoder is used to obtain features of multiple scales through deep coding;

[0037] A dynamic upsampling unit, configured to calculate a bias constant based on the features of the multiple different scales, calculate a sampling group based on the bias constant, and obtain a feature map retaining local detail features after dynamic upsampling through bilinear interpolation;

[0038] The disparity head unit is used to obtain a depth map by generating a disparity map based on the feature map.

[0039] Another aspect of the present invention provides an electronic device, comprising: one or more processors and a memory, wherein the memory stores one or more programs, and the one or more programs include instructions for executing the aforementioned lightweight monocular depth estimation method based on self-supervised deep learning.

[0040] Another aspect of the present invention provides a computer-readable storage medium comprising one or more programs for execution by one or more processors of an electronic device, wherein the one or more programs include instructions for executing the aforementioned lightweight monocular depth estimation method based on self-supervised deep learning.

[0041] Compared with the prior art, the present invention has at least one of the following beneficial effects:

[0042] (1) Fully retaining detail features during upsampling: In model training, the present invention obtains a feature map that retains local detail features based on training data through multi-scale feature extraction and dynamic upsampling, and obtains a depth map based on the feature map. By introducing efficient dynamic upsampling, more detailed local features are retained, thereby obtaining a more accurate depth estimation result.

[0043] (2) Good performance in gradient-rich areas: During the training phase, the present invention performs frame reconstruction based on relative pose information and depth maps, calculates the loss function value under the premise of considering the semantically-aware triplet loss, and implements the training of the depth estimation model. By adding the triplet loss, the model performance can be significantly improved in gradient-rich areas without affecting the inference speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 It is a flowchart of the training phase of the lightweight monocular depth estimation method based on self-supervised deep learning in an embodiment;

[0045] Figure 2 is a schematic diagram of the overall structure of a lightweight monocular depth estimation model based on self-supervised deep learning in an embodiment;

[0046] Figure 3 Schematic diagram of the structure of the efficient dynamic upsampling module in the embodiment;

[0047] Figure 4 is a schematic diagram of the principle of the improved semantic-aware triplet loss in the embodiment;

[0048] Figure 5 3 is a depth estimation effect diagram of the lightweight monocular depth estimation method based on self-supervised deep learning in the embodiment. DETAILED DESCRIPTION

[0049] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0050] Example 1

[0051] In view of the problems existing in the aforementioned prior art, this embodiment provides a lightweight monocular depth estimation method based on self-supervised deep learning, which provides a lightweight but reliable self-supervised monocular depth estimation architecture. The architecture uses an efficient dynamic upsampling module (EDU Module) to retain detailed features. In addition, the architecture also incorporates an improved semantically-aware triplet loss (SaTri Loss) in the loss calculation to improve the estimation accuracy in gradient-rich areas.

[0052] This method mainly includes two parts: training phase and testing phase.

[0053] See also Figure 1 The flowchart of the training phase includes the following steps:

[0054] Step S1, obtaining training data: capturing multiple frames of continuous RGB images in a monocular video sequence as a training sample to form training data.

[0055] Step S2: The camera pose estimation network estimates the relative pose of the camera: Figure 2 In the camera pose estimation network (Pose Estimation Network), the training data is extracted features through a pose encoder (this embodiment is implemented using ResNet1 8), and then the extracted features are sent to a pose decoder to predict the relative pose of the camera between two frames.

[0056] Step S3, the depth estimation network performs depth estimation: see Figure 2 In the depth estimation network, the training data is extracted features at multiple (three in this embodiment) different scales through a depth encoder (i.e., Feature Extraction Network), and then sent to a depth decoder. The training data is upsampled to retain more local detail features through an efficient dynamic upsampling module (EDU Module). Finally, a disparity map is generated based on the upsampled feature map using a disparity head to obtain a depth map.

[0057] In the depth estimation network decoder, the efficient dynamic upsampling module takes the features extracted by the depth encoder as input. The module first uses two linear layers to calculate a bias constant, then calculates the sampling group through the bias constant and the original sampling coordinates, and finally obtains the upsampled feature map through bilinear interpolation (Bilinear Interpolation Upsample). The module structure is as follows Figure 3 The specific calculation is as follows:

[0058]

[0059]

[0060]

[0061] in is the input feature map of size C×H×W, The final feature map of size C×sH×sW obtained by upsampling (s is the sampling ratio, s=2 in this embodiment), Linear a With Linear b For calculating the bias constant The two linear layers have input and output dimensions of C and 2s2 respectively. is the original sampling coordinate, For calculation and The sampling set obtained by summing up, grid_sample is the input feature map according to the given coordinate grid S The sampling function is a built-in function of PyTorch.

[0062] Step S4, network training: using the depth map obtained in step S3 and the relative camera pose predicted in step S2, reconstruct the current (middle) frame image from adjacent (front and back) frames, and use the photometric reprojection loss to constrain the consistency of the reconstructed frame and the target frame; combine the photometric reprojection loss, edge smoothing loss and improved semantically-aware triplet loss as the final training loss.

[0063] The depth estimation map and the predicted camera motion are used to reconstruct the current frame image from the adjacent frames, and the photometric reconstruction loss is used to constrain the consistency between the reconstructed frame and the target frame. The photometric reconstruction loss, edge smoothing loss and improved semantic-aware triplet loss are combined as the final training loss.

[0064] Among them, the improved semantic-aware triplet loss principle is as follows Figure 4 As shown, the calculation method is as follows:

[0065]

[0066]

[0067]

[0068] Among them, after the image is segmented into triplets according to semantic information, represents the central pixel (anchor point), is a point with the same semantic information as the anchor point (positive sample), are other points with different semantic information from the anchor point (negative samples), and Represent the distance between anchor and positive sample and anchor and negative sample respectively, F d represents the corresponding deep features, is a set of all semantic boundary pixels that meet the above conditions.

[0069] Testing phase: Take three consecutive frames in the video sequence as a test sample, and the middle frame as the depth estimation target. Send the test sample to the depth estimation network, and the output of the final depth estimation network is the estimated depth map of the target frame. Figure 5 As shown, (a) is the input image, (b) is the result of the Lite-Mono method for comparison, and (c) is the result of the method of this embodiment. Figure 5It can be seen that the target edge in the depth estimation image corresponding to this method is smooth, and the depth prediction accurately reflects the scene depth information, which is consistent with human vision.

[0070] Example 2

[0071] Based on Example 1, this embodiment provides a lightweight monocular depth estimation device based on self-supervised deep learning, including:

[0072] The pose estimation module corresponds to the camera pose estimation network in Example 1, and is used to predict the relative pose information of two adjacent frames of images through pose encoding and decoding based on the input of multiple consecutive frames of images.

[0073] The depth estimation module corresponds to the depth estimation network in Example 1, and is used to obtain a feature map that retains local detail features based on continuous multi-frame images through multi-scale feature extraction and dynamic upsampling, and obtain a depth map based on the feature map. Figure 2 , the depth estimation module includes:

[0074] A deep encoder is used to obtain features of multiple scales through deep coding;

[0075] A dynamic upsampling unit is used to calculate a bias constant based on features of multiple different scales, calculate a sampling group based on the bias constant, and obtain a feature map retaining local detail features after dynamic upsampling through bilinear interpolation;

[0076] The disparity head unit is used to obtain a depth map by generating a disparity map based on the feature map.

[0077] The training module is used to reconstruct frames based on relative pose information and depth maps during the training phase, calculate the loss function value under the premise of considering the semantically-aware triplet loss, and realize the training of the pose estimation module and the depth estimation module.

[0078] To verify the effectiveness of this method (LiDUT-Depth), we compare it with the existing work performance. See Table 1 for a schematic diagram of the work performance comparison under the Eigen split of KITTI, where all input images are resized to 640×192. The first and second ranked results are bolded and underlined, respectively. Table 2 shows the generalization performance test data of different datasets, which compares the performance of our method and existing work on the Make3D dataset. All models are trained on KITTI with a resolution of 640×192. Table 3 shows the ablation experiment results. All models are trained on KITTI with an input resolution of 640×192.

[0079] Table 1 Comparison of working efficiency

[0080]

[0081] Table 2 Generalization performance test data

[0082]

[0083] Table 3 Ablation experiment results

[0084]

[0085] It can be seen from Tables 1 to 3 that, compared with existing work, the method provided in this embodiment has the advantages of high estimation accuracy and small model space occupation.

[0086] The above method and device have the following beneficial effects: a lightweight self-supervised monocular depth estimation architecture is provided to further improve the accuracy of depth estimation without increasing computational overhead or increasing model size, thereby improving the reliability and deployability of the depth estimation model. An efficient dynamic upsampling module is introduced to retain more detailed local features to obtain more accurate depth estimation results. The improved semantic-aware triplet loss is integrated into the training loss to further optimize the model and significantly improve the model performance in gradient-rich areas without affecting the inference speed.

[0087] Example 3

[0088] This embodiment provides an electronic device, including: one or more processors and a memory, wherein one or more programs are stored in the memory, and the one or more programs include instructions for executing the lightweight monocular depth estimation method based on self-supervised deep learning described in Example 1.

[0089] Example 4

[0090] This embodiment provides a computer-readable storage medium, including one or more programs for execution by one or more processors of an electronic device, and the one or more programs include instructions for executing the lightweight monocular depth estimation method based on self-supervised deep learning as described in Example 1.

[0091] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A lightweight monocular depth estimation method based on self-supervised deep learning, characterized in that: Acquire multiple consecutive frames of images, and reconstruct the target frame based on adjacent frames of the target frame using a trained depth estimation model to obtain depth information of the target frame, wherein the training process of the depth estimation model includes the following steps: Obtain a monocular video sequence and construct training data by extracting multiple consecutive frames of images; Based on the training data, relative pose information of two adjacent frames of images is predicted through pose encoding and decoding; Based on the training data, a feature map retaining local detail features is obtained through multi-scale feature extraction and dynamic upsampling, and a depth map is obtained based on the feature map; Frame reconstruction is performed based on the relative pose information and the depth map, and a loss function value is calculated under the premise of considering a semantically-aware triplet loss to implement training of the depth estimation model.

2. According to claim 1, a lightweight monocular depth estimation method based on self-supervised deep learning is characterized in that: The process of obtaining a feature map retaining local detail features based on the training data through multi-scale feature extraction and dynamic upsampling, and obtaining a depth map based on the feature map comprises the following steps: Based on the training data, obtaining features of multiple scales through deep coding; Calculating a bias constant based on the features of the multiple different scales, calculating a sampling group based on the bias constant, and obtaining a feature map retaining local detail features after dynamic upsampling through bilinear interpolation; Based on the feature map, a depth map is obtained by generating a disparity map.

3. According to a lightweight monocular depth estimation method based on self-supervised deep learning according to claim 1, it is characterized in that: The dynamic upsampling is achieved by the following formula: in, The feature map of size C×H×W after multi-scale feature extraction of the input is The final feature map of size C×sH×sW obtained by dynamic upsampling, s is the sampling ratio, Linear a With Linear b For calculating the bias constant The linear layer has input and output dimensions of C and 2s respectively. 2 , is the original sampling coordinate, For calculation and The sample set obtained by summing According to the given coordinate grid For the input feature map The function to perform sampling.

4. The lightweight monocular depth estimation method based on self-supervised deep learning according to claim 1, characterized in that: The calculation process of the semantically-aware triplet loss includes: Based on the reconstructed image of the frame, the image is segmented into anchor points, positive samples and negative samples through semantic segmentation; By calculating the anchor-positive sample distance and the anchor-negative sample distance, the triplet loss is obtained.

5. The lightweight monocular depth estimation method based on self-supervised deep learning according to claim 1, characterized in that: The semantically-aware triplet loss is obtained using the following formula: in, Represents the anchor point, is a point with the same semantic information as the anchor point, i.e., a positive sample. are other points with different semantic information from the anchor point, i.e. negative samples. and Represent the distance between anchor point and positive sample and the distance between anchor point and negative sample, respectively. d represents the corresponding deep features, is the set of semantic boundary pixels, m ′ is the threshold, which is used to adjust the minimum interval between the two to prevent a large gap.

6. The lightweight monocular depth estimation method based on self-supervised deep learning according to claim 1, characterized in that: The loss function value also includes photometric reprojection loss and edge smoothing loss.

7. A lightweight monocular depth estimation device based on self-supervised deep learning, characterized in that: include: The pose estimation module is used to predict the relative pose information of two adjacent frames of images through pose encoding and decoding based on the input of multiple consecutive frames of images; A depth estimation module, configured to obtain a feature map retaining local detail features based on the continuous multi-frame images through multi-scale feature extraction and dynamic upsampling, and obtain a depth map based on the feature map; A training module is used to perform frame reconstruction based on the relative pose information and the depth map during the training phase, calculate the loss function value under the premise of considering the triplet loss based on semantic perception, and realize the training of the pose estimation module and the depth estimation module.

8. The lightweight monocular depth estimation device based on self-supervised deep learning according to claim 7, characterized in that: The depth estimation module comprises: A deep encoder is used to obtain features of multiple scales through deep coding; A dynamic upsampling unit, configured to calculate a bias constant based on the features of the multiple different scales, calculate a sampling group based on the bias constant, and obtain a feature map retaining local detail features after dynamic upsampling through bilinear interpolation; The disparity head unit is used to obtain a depth map by generating a disparity map based on the feature map.

9. An electronic device, characterized in that: include: One or more processors and a memory, wherein the memory stores one or more programs, and the one or more programs include instructions for executing the lightweight monocular depth estimation method based on self-supervised deep learning as described in any one of claims 1-6.

10. A computer-readable storage medium, characterized in that: It includes one or more programs for execution by one or more processors of an electronic device, and the one or more programs include instructions for executing a lightweight monocular depth estimation method based on self-supervised deep learning as described in any one of claims 1-6.

Images