An automatic driving-oriented structured scene depth estimation method
By constructing a dual-encoder-quad-decoder network and an improved loss function, and utilizing RGB images and millimeter-wave radar data, the problem of insufficient depth estimation accuracy in autonomous driving scenarios is solved, and higher depth estimation accuracy is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- UNIV OF ELECTRONICS SCI & TECH OF CHINA
- Filing Date
- 2023-05-24
- Publication Date
- 2026-06-09
Smart Images
Figure CN116485867B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of autonomous driving technology, specifically to a structured scene depth estimation method for autonomous driving. Background Technology
[0002] Monocular depth estimation is a long-standing ill-posed problem in the field of computer vision. It uses a single RGB image to estimate the distance from each point in the scene to the camera and has wide applications in many fields such as robotics, autonomous driving, and 3D reconstruction.
[0003] Traditional monocular depth estimation methods primarily utilize hand-designed features, with representative methods including Structure of Motion (SFM) and methods based on traditional machine learning. SFM uses camera motion as a cue for depth estimation, while traditional machine learning methods build a model between the image and depth using Markov Random Fields (MRFs) or Conditional Random Fields (CRFs), learning the mapping relationship between input features and output depth to obtain depth estimation information.
[0004] In recent years, deep neural networks have developed rapidly and have demonstrated excellent performance in image processing tasks such as image classification, image detection, and image segmentation. Therefore, researchers have introduced them into monocular depth estimation. In 2014, Eigen et al. first used deep convolutional neural networks for monocular depth estimation. It used RGB images as input and a two-stage network to coarsely predict global image information and finely refine local image information. Since deep learning was applied to monocular depth estimation, related methods have been continuously improved, such as building multi-scale networks to improve performance, using encoder-decoder structures for depth estimation, or stratifying depth estimation from a regression task to a classification task. The training of these methods all rely on the real depth labels of the scene. Due to the high cost of pixel-by-pixel annotation, unsupervised learning methods have also received widespread attention. These methods typically use paired stereo images or image sequences for training, supervising the network training through image reconstruction loss, thus avoiding the large human resource investment required in the annotation process.
[0005] Depth completion tasks incorporate depth sensors, such as LiDAR and millimeter-wave radar, to reconstruct dense depth maps from coarse depth maps obtained from these sensors. While purely visual depth estimation methods have achieved satisfactory results, fusing additional depth information from sensors with RGB image information significantly improves accuracy. The key challenges of depth completion lie in the sparse and noisy input depth map and how to effectively integrate information from both the image and depth dimensions to obtain better results. Current depth completion methods utilize multi-branch networks, using encoders to extract features from the sparse depth map and its corresponding RGB image, then fusing these features at different levels, and finally decoding to obtain a dense depth map. With advancements in depth completion technology, surface normals and affinity matrices have been incorporated into network models, further contributing to its development.
[0006] In autonomous driving scenarios, depth estimation plays a crucial role. Depth estimation in structured scenes possesses relatively standard scene characteristics; however, past methods have not considered leveraging scene information to improve depth estimation predictions, nor have they fully utilized semantic information within the scene. Therefore, it is necessary to improve existing methods. Knot This study aims to improve the depth estimation method for structured scenes in order to enhance the accuracy of depth estimation. Summary of the Invention
[0007] The purpose of this invention is to address the shortcomings of existing depth estimation methods by proposing a structured scene depth estimation method for autonomous driving. This method uses RGB images and sparse depth maps as inputs to construct a dual-encoder-quad-decoder network structure based on semantic information within the scene, thereby improving the accuracy of depth estimation. In constructing the dual-encoder-quad-decoder network structure, an improved L1 loss function is designed, assigning different weights to different target categories in the scene to enhance network performance.
[0008] To achieve the above objectives, the present invention adopts the following technical solution:
[0009] A structured scene depth estimation method for autonomous driving includes the following steps:
[0010] Step 1: Design a dual-encoder-quad-decoder network
[0011] The dual-encoder-quad-decoder network consists of a dual-encoder network and a quad-decoder network. The dual-encoder network takes RGB images and millimeter-wave radar data as inputs, extracts features from them respectively, and then fuses them to obtain the first fused feature map.
[0012] The four-decoder network consists of four decoders: the first decoder, the second decoder, the third decoder, and the fourth decoder. The first decoder is a segmentation decoder, while the second, third, and fourth decoders are all depth decoders. First, a first fused feature map is input into each of the four decoders. The first decoder decodes the first fused feature map to generate a semantic segmentation map and then divides the scene into three feature categories: road and traffic participant features, tree and building features, and sky features. The three depth decoders decode the received first fused feature map, each obtaining an initial predicted depth map. These three initial predicted maps are then fused one-to-one with the three feature categories to obtain depth maps for different scene categories. Finally, the depth maps for different scene categories are fused to obtain the final predicted depth map.
[0013] Step 2: Design the loss function of the dual-encoder-quad-decoder network
[0014] The loss function of a dual-encoder-quad-decoder network consists of four parts, namely the depth loss L... depth Smoothing loss L smooth The supervision loss L for the feature maps generated by the sparse pre-module map Supervision loss L for semantic segmentation results seg Among them, depth loss L depth It is an improved function based on L1 loss, which assigns different weights to roads and traffic participants, trees and buildings, and the sky in the scene;
[0015] Depth loss L depth As shown in equation (2):
[0016]
[0017] In equation (2), d and Let S1 and S2 represent the true depth map and the predicted depth map, respectively. S1 represents the set of features in d that belong to the road and traffic participants, and S2 represents the set of features in d that do not belong to the road and traffic participants. m is the number of effective depths, and ω is the hyperparameter that needs to be adjusted. When ω is 1.4, the balance between feature points of different categories in the autonomous driving scenario is optimal.
[0018] Smoothing loss L smooth As shown in equation (3):
[0019]
[0020] In equation (3), Let x and y represent the gradients along the x and y directions, respectively, and I represent the input image.
[0021] The loss function of the complete dual-encoder-quad-decoder network is shown in equation (4):
[0022] L total =λ1(L depth +λ2L smooth +λ3L map )+L seg (4)
[0023] In equation (4), λ1, λ2, and λ3 are all weighting factors, which are set based on experience;
[0024] Step 3: Supervise the network using depth labels and segmentation labels as ground truth, and use the loss function obtained in Step 2 as feedback to train the dual-encoder-quad-decoder network;
[0025] Step 4: Input the RGB image to be estimated and the millimeter-wave radar data into the trained dual-encoder-quad-decoder network to perform depth estimation of the scene and obtain the final predicted depth map.
[0026] Furthermore, the nuScenes dataset was used in the construction and training of the dual-encoder-quad-decoder network.
[0027] Furthermore, the dual-encoding network includes an image encoder and a depth encoder; wherein the image encoder is a pre-trained ResNet-34 network with fully connected layers removed; the depth encoder includes a sparse pre-mapping module and a residual module, wherein the sparse pre-mapping module extracts preliminary features from the millimeter-wave radar data, and the residual module further extracts features.
[0028] Furthermore, the depth decoder consists of four sequentially connected upsampling modules. Based on the input first fused feature map, it first generates a 16-channel feature map with a resolution half that of the input image. Then, it maps the generated feature map to a single channel through a 3×3 convolution. Finally, it uses bilinear upsampling to the original resolution and outputs it directly as the initial prediction map.
[0029] Furthermore, the segmentation decoder is similar in structure to the depth decoder, except that it maps the generated features to nineteen channels of different segmentation categories through 3×3 convolution, and then uses the softmax function to classify them to obtain three feature category outputs.
[0030] This invention provides a structured scene depth estimation method for autonomous driving, which uses RGB images and sparse depth maps as inputs to construct a dual-encoder-four-decoder network structure based on semantic information in the scene. This network structure uses dual encoders to process the input RGB image and millimeter-wave radar data, employing a sparse pre-mapping module to extract sparse millimeter-wave radar features and fuse them with image features to obtain a first fused feature map. The first fused feature map is then decoded by four decoders. During decoding, one decoder is used to decode the first fused feature map to obtain a semantic segmentation map, which is used to classify the scene into three feature categories. The other three depth decoders predict the depth maps of the three types of targets in the scene, i.e., each decoder decodes the first feature fused map to obtain an initial prediction map. The three initial prediction maps are fused one-to-one with the three feature categories, thereby introducing semantic information from the scene. Combined with the improved L1 loss function designed in this invention, which assigns different weights to different categories of targets in the scene based on L1 loss to improve network performance.
[0031] Compared with existing technologies, the present invention has higher accuracy in depth estimation. Attached Figure Description
[0032] Figure 1 This is a schematic diagram of a dual-encoder-quad-decoder network architecture as an example.
[0033] Figure 2 This is a schematic diagram of the sparse pre-mapping module in an embodiment;
[0034] Figure 3 This example illustrates the depth map fusion process under different scene categories.
[0035] Figure 4 This is a schematic diagram illustrating the training and derivation of the dual-encoder-quad-decoder network in this embodiment;
[0036] Figure 5 The image shows the depth estimation results obtained in the example. Detailed Implementation
[0037] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.
[0038] This embodiment provides a structured scene depth estimation method for autonomous driving, including the following steps:
[0039] Step 1: Design a dual-encoder-quad-decoder network
[0040] like Figure 1 As shown, the dual-encoder-quad-decoder network consists of a dual-encoder network and a quad-decoder network.
[0041] The dual-encoding network includes an image encoder and a depth encoder. The image encoder is a ResNet-34 network pre-trained on ImageNet with fully connected layers removed. It includes four sequentially connected convolutional modules, which generate feature maps of 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the original image size, respectively, in the order of connection. The number of channels in the four convolutional modules, in the order of connection, are 64, 128, 256, and 512, respectively.
[0042] The depth encoder includes a sparse pre-mapping module and a residual module. The sparse pre-mapping module extracts preliminary features from the millimeter-wave radar data, and the residual module further extracts features. Figure 2 As shown, the sparse pre-mapping module obtains a denser feature map through five stacked sparse invariant convolutions. After bilinear upsampling to the original resolution at the output, supervision is applied to this output. The sparse invariant convolutions use progressively decreasing kernel sizes: 7, 5, 3, 3, 1. The first four convolutions have 16 output channels, and the last convolution has 1. The stride of the first convolution is 2, and the stride of the remaining convolutions is 1, all to obtain a denser output for supervision. Finally, the output of the fourth convolution is used as the input to the residual module, which further extracts higher-level features. In this embodiment, the calculation formula used by the sparse pre-mapping module is:
[0043]
[0044] In equation (1), x is the input; o represents the binary value 1 or 0 corresponding to the input x (1 indicates that there is an observation) or 0 indicates that there is no observation; W represents the weight parameter; b represents the bias; u and v are the pixel coordinates; ε is a very small positive number to prevent division by zero;
[0045] The residual module uses four convolutional modules with fewer layers. The feature maps obtained by the four convolutional modules along the output direction are respectively 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the original image size, and have 16, 32, 64, and 128 channels respectively.
[0046] The quad-decoder network consists of four decoders: the first decoder, the second decoder, the third decoder, and the fourth decoder. The first decoder is a segment decoder, while the second, third, and fourth decoders are all depth decoders.
[0047] First, the first fused feature map is input into four decoders. The first decoder generates the semantic segmentation map and contains four sequentially connected upsampling modules. After passing through these four upsampling modules, the input first fused feature map yields feature maps of 1 / 16, 1 / 8, 1 / 4, and 1 / 2 of the original image size, respectively. The number of channels in the four upsampling modules are 128, 64, 32, and 16, respectively. The output of the last upsampling module is bilinearly upsampled to 19 channels and then classified using a softmax function to obtain the final segmentation result, resulting in three feature categories: road and traffic participant features, tree and building features, and sky features. The three deep decoders have similar structures to the segmentation decoders, also containing four sequentially connected upsampling modules. However, the output of the last upsampling module in the deep decoder is bilinearly sampled to the original resolution and then directly used as the initial prediction map. Figure 3 As shown, the three initial prediction maps generated by the three depth decoders are fused one-to-one with the three feature categories to obtain depth maps under different scene categories; then the depth maps under different scene categories are fused to obtain the predicted depth map.
[0048] Step 2: Design the loss function for the dual-encoder-quad-decoder network. The loss function consists of four parts, namely the depth loss L... depth Smoothing loss L smooth The supervision loss L for the feature maps generated by the sparse pre-module map Supervision loss L for semantic segmentation results seg This includes the following sub-steps:
[0049] 2.1 Improved L1 loss
[0050] In autonomous driving scenarios, there are certain relationships between pixels. When optimizing network parameters, it is necessary to consider the balance between different categories of points and design an appropriate loss function. Based on this, this implementation assigns different weights to roads and traffic participants, trees and buildings, and the sky in the scene. Based on L1 loss, the depth loss function is designed as follows:
[0051]
[0052] In equation (2), d and Let S1 and S2 represent the ground truth depth map and the predicted depth map, respectively. S1 represents the set of objects in d that belong to roads and traffic participants, and S2 represents the set of objects in d that do not belong to roads and traffic participants. m is the number of effective depths, and ω is the hyperparameter that needs to be adjusted.
[0053] Extensive experiments demonstrate that a suitable parameter ω can balance the pixel counts of different categories in the scene, further improving optimization performance during training. Different values of ω were tested, starting from 0 and using a step size of 0.2, to obtain the errors for each category, as shown below.
[0054] As shown in Table 1:
[0055]
[0056] It is easy to see from the table that when ω is 1.4, the balance between feature pixels of each category in the autonomous driving scenario is optimal, and the prediction effect is the best.
[0057] 2.2 Definition of Smoothing Loss
[0058] Since depth discontinuities typically occur at boundaries, image gradients are used for weighting, and a smoothing loss L is applied. sm oo th Defined as:
[0059]
[0060] in These represent the gradients along the x and y directions, respectively. I represents the input image.
[0061] 2.3 Introducing supervised loss, which consists of two parts: one is the supervised loss for the depth map generated by the sparse pre-mapping module, denoted as L. map Second, the supervision loss introduced into the semantic segmentation result of the segmentation decoder, denoted as L. seg
[0062] Therefore, the loss function of the dual-encoder-quad-decoder network is:
[0063] L total =λ1(L depth +λ2L smooth +λ3L map )+L seg (4)
[0064] λ1, λ2, and λ3 are hyperparameters set based on experience.
[0065] Step 3: Supervise the network using depth labels and segmentation labels as ground truth, and use the loss function obtained in Step 2 for feedback to train the dual-encoder-quad-decoder network. For example... Figure 4 As shown, this implementation uses only images and millimeter-wave radar as input to generate depth maps during training.
[0066] Step 4: Input the RGB image to be estimated and the millimeter-wave radar data into the trained dual-encoder-quad-decoder network to perform depth estimation of the scene. The results are as follows: Figure 5As shown in the figure. The color gradient from blue to red in the prediction results indicates an increase in depth value, with the estimated maximum depth value being 120 meters.
[0067] This embodiment uses the nuScenes dataset for both training and testing of the dual-encoder-quad-decoder network. The nuScenes dataset contains not only camera and LiDAR data but also millimeter-wave radar data, making it one of the few large datasets containing millimeter-wave radar data. Each scene in this dataset is 20 seconds long, with 40 keyframes, and each frame has a resolution of 1600×900. Furthermore, nuScenes includes various driving scenarios, such as rainy days and nighttime conditions, which increases the difficulty of depth estimation on this dataset. This invention uses 850 scenes, dividing them into 810 scenes for training and 40 scenes for evaluation (the training set contains 32,564 images, and the test set contains 1,585 images). The final estimated depth map estimates the final depth at all 1.44 million pixels, achieving a density improvement of approximately 20,000 times compared to the initial millimeter-wave radar detecting only 40-50 effective points per frame.
[0068] This embodiment uses PyTorch to deploy the network and trains it on an NVIDIA GeForce GTX TITAN X. The batch size is set to 4, and the Adam optimizer with a learning rate of 0.0005 is used, halving every 5 epochs. The parameters are set to λ1 = 0.5, λ2 = 0.001, and λ3 = 0.3. The results are calculated at all pixel locations, as shown in Table 2. It can be seen that the various metrics of this invention are superior to existing state-of-the-art solutions, demonstrating that performing depth estimation separately for different categories and using the loss function proposed in this invention effectively improves the network performance. Let d and Let represent the predicted depth map and label, respectively; n represents the number of observation points with LiDAR depth values in each image; and Y represents the measurement range. The evaluation metrics used are shown below:
[0069] Root Mean Square Error (RMSE):
[0070] Mean Absolute Error (MAE):
[0071] Table 2 Depth Estimation Results
[0072]
Claims
1. A method for estimating depth of structured scene for autonomous driving, characterized in that, Includes the following steps: Step 1: Design a dual-encoder-quad-decoder network The dual-encoder-quad-decoder network consists of a dual-encoder network and a quad-decoder network. The dual-encoder network includes an image encoder and a depth encoder. It takes RGB image and millimeter-wave radar data as input, extracts features from them respectively, and then fuses them to obtain a first fused feature map. The depth encoder includes a sparse pre-mapping module and a residual module. The sparse pre-mapping module is used to extract preliminary features from the millimeter-wave radar data, and then the residual module is used to further extract features. The four-decoder network consists of four decoders: the first decoder, the second decoder, the third decoder, and the fourth decoder. The first decoder is a segmentation decoder, while the second, third, and fourth decoders are all deep decoders. First, the first fused feature map is input into the four decoders. The first decoder decodes the first fused feature map to generate a semantic segmentation map and divides the scene into three feature categories based on the semantic segmentation map: road and traffic participant features, tree and building features, and sky features. Three depth decoders decode the received first fused feature map respectively, each obtaining an initial predicted depth map; the three initial predicted maps are fused one-to-one with the three feature categories to obtain depth maps under different scene categories; then the depth maps under different scene categories are fused to obtain the predicted depth map. Step 2: Design the loss function of the dual-encoder-quad-decoder network The loss function of the dual-encoding and quadruple-decoding network is composed of four parts, namely, a depth loss , a smoothing loss , a supervision loss for the feature map generated by the sparse pre-module , and a supervision loss for the semantic segmentation result ; wherein the depth loss is an improved function based on L1 loss, which respectively gives different weights to roads and traffic participants, trees and buildings, and sky in the scene. depth loss As shown in equation (2): In equation (2), and These represent the true depth map and the predicted depth map, respectively. express The middle part refers to the collection of road and traffic participants. express The set of features that do not belong to roads and traffic participants, m is the number of effective depths, and ω is the hyperparameter that needs to be adjusted; when ω takes the value of 1.4, the balance between feature points of each category in the autonomous driving scenario reaches the optimal level. Smoothing loss As shown in equation (3): In equation (3), Let x and y represent the gradients along the x and y directions, respectively. Indicates the input image; The loss function of the complete dual-encoder-quad-decoder network is shown in equation (4): In equation (4), , , These are all weighting factors, set based on experience; Step 3: Supervise the network using depth labels and segmentation labels as ground truth, and use the loss function obtained in Step 2 as feedback to train the dual-encoder-quad-decoder network; Step 4: Input the RGB image to be estimated and the millimeter-wave radar data into the trained dual-encoder-quad-decoder network to perform depth estimation of the scene and obtain the final predicted depth map.
2. The structured scene depth estimation method for autonomous driving according to claim 1, characterized in that: The nuScenes dataset was used in both the construction and training of the dual-encoder-quad-decoder network.
3. The structured scene depth estimation method for autonomous driving according to claim 1, characterized in that: The image encoder is a pre-trained ResNet-34 network with fully connected layers removed.
4. The structured scene depth estimation method for autonomous driving according to claim 1, characterized in that: The depth decoder consists of four sequentially connected upsampling modules. Based on the first fused feature map of the input, it first generates a 16-channel feature map with a resolution half that of the input image. Then, it maps the generated feature map to a single channel through a 3×3 convolution. Finally, it uses bilinear upsampling to the original resolution and outputs it directly as the initial prediction map.
5. The structured scene depth estimation method for autonomous driving according to claim 4, characterized in that: The segmentation decoder is similar in structure to the depth decoder, but the difference is that it uses 3×3 convolution to map the generated features to nineteen channels of different segmentation categories, and then uses the softmax function to classify them to obtain three feature category outputs.