A real-time depth estimation method based on semantics and multi-scale information

Abstract

The present application belongs to the technical field of automatic driving, and specifically relates to a real-time depth estimation method based on semantics and multi-scale information. During training, the feature distillation method is used to migrate the prediction result of the pre-trained semantic segmentation network to the depth estimation network, so as to force the depth estimation network to learn semantic information during the training process. After the training is completed, the semantic segmentation network is removed, and the same performance as the multi-task architecture is realized in the single-branch structure. Through the exploration of the role of multi-scale information in the depth estimation task, it is assumed that the larger scale feature is suitable for the estimation of the target edge, and the small scale feature is suitable for the estimation of the large range area inside the target, and a multi-scale fusion module is designed. Through the multi-scale fusion module, the decoder features of different scales are mapped into the corresponding scale depth map and confidence map, and are subjected to up-sampling reconstruction and fusion to obtain the final depth map, so that the estimation accuracy is improved.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving technology, specifically to a real-time depth estimation method based on semantic and multi-scale information. Background Technology

[0002] Autonomous vehicles require a deep understanding of their 3D environment. In purely vision-based solutions, the missing depth information in the image can be estimated using stereo or monocular images. Stereo-based solutions can obtain relatively accurate depth maps, but they require calibrated binocular cameras. Monocular-based solutions estimate depth from only a single image, and their ease of deployment and lower cost have attracted widespread attention.

[0003] Monocular depth estimation is an ill-posed problem, as monocular images themselves do not provide any effective depth information. Therefore, some works have significantly improved the accuracy of monocular depth estimation by incorporating depth information from lidar, given the known depth values ​​of some pixels. However, lidar, as an expensive sensor, presents a significant challenge to systems with real-time requirements due to its large amount of point cloud data. To address this issue, some works have replaced lidar with sparse millimeter-wave radar information to assist in monocular depth estimation. In recent years, many works have explored depth estimation tasks, such as introducing attention mechanisms or transforming depth estimation from a regression task into a classification task. Unlike generalized networks designed for all scenarios, driving scenarios are highly structured, with fixed components, each possessing unique characteristics. For example, the depth values ​​of ground areas typically change slowly, and the depth values ​​of pedestrians and vehicles on roads are usually similar to those of the surrounding ground. The depth values ​​of the sky, however, are often very large, even unpredictable. Furthermore, large changes in depth values ​​usually only occur in semantically variable regions. Based on this characteristic, some works have adopted a multi-task architecture for the network, simultaneously training depth estimation and semantic segmentation tasks within the network. Depth estimation is achieved by coordinating the weights of these two tasks. While this approach can achieve depth estimation, finding suitable weights in practice requires extensive experimentation, and even then, finding the optimal weights is not guaranteed. Furthermore, the two tasks typically involve direct architectural connections, making it difficult to maintain a high degree of simplicity in the network architecture during derivation. Summary of the Invention

[0004] The purpose of this invention is to provide a real-time depth estimation method based on semantic and multi-scale information. This network can achieve performance comparable to complex networks with a low-complexity single-stage architecture and reach a frame rate of 50 FPS.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] A real-time depth estimation method based on semantic and multi-scale information includes the following steps:

[0007] Step 1: Construct a real-time depth estimation network based on semantic and multi-scale information.

[0008] Real-time depth estimation network based on semantic and multi-scale information, including a dual-branch initial feature extraction network, a shared encoder and a decoding network;

[0009] A dual-branch initial feature extraction network takes RGB image and millimeter-wave radar data as input, extracts features from them respectively, and then fuses them to obtain the first fused feature output.

[0010] The shared encoder receives the first fused feature and abstracts it to obtain the encoder feature output;

[0011] The decoding network includes a semantic segmentation network and a depth estimation network. The semantic segmentation network is a pre-trained network structure used to provide semantic information of the scene for the depth estimation network during training. The deep network includes a depth decoder and a multi-scale information fusion module. The input of the depth decoder is the encoder features, which are used to decode the encoder features to obtain decoder feature outputs at different scales. The multi-scale information fusion module has a mapping structure and an upsampling unit. The mapping structure receives decoder features at different scales and maps them into depth maps and confidence maps of the corresponding scales. After reconstruction by the upsampling unit, they are fused to obtain the final depth map.

[0012] Step 2: Design the loss function for a real-time depth estimation network based on semantic and multi-scale information.

[0013] The loss function of the real-time depth estimation network based on semantic and multi-scale information consists of two parts: semantic information transfer-related loss and depth estimation-related loss.

[0014] The loss related to semantic information transfer is: the feature distillation loss generated by transferring semantic information to the deep estimation network using feature distillation, as shown in Equation (5):

[0015]

[0016] In equation (5), F s F represents the features of a semantic segmentation network. d The features of the deep estimation network are represented by two features with the same scale.

[0017] The depth estimation loss consists of three components: the error between the final depth map and the ground truth. As shown in equation (6); smoothing constraint As shown in equation (7); the error of the depth map in the multi-scale information fusion module As shown in equation (8);

[0018]

[0019] In equation (6), T is the set of all pixels in the image, m is the total number of valid pixels (i.e., pixels with a true value greater than 0), where p i and y i These represent the predicted depth value and the actual depth value of pixel i, respectively.

[0020]

[0021] In equation (7), and This indicates taking the derivative with respect to the x and y directions;

[0022]

[0023] In equation (8), The loss term represents the loss for different intermediate depth maps, P = 4;

[0024] Therefore, the complete loss function of the designed real-time depth estimation network based on semantic and multi-scale information is shown in equation (9):

[0025]

[0026] In equation (9), ω1, ω2 and ω3 are hyperparameters, and ω1 = 0.01, ω2 = 0.5 and ω3 = 4;

[0027] Step 3: Train a real-time depth estimation network based on semantic and multi-scale information. The training process includes the following sub-steps:

[0028] 3.1. Train the semantic segmentation network based on the encoder features and truth map output by the shared coding network, obtain the semantic segmentation network weights, and fix their weights in subsequent training;

[0029] 3.2 Transfer the semantic segmentation information to the depth estimation network, use the loss function obtained in step 2 for feedback, and train the real-time depth estimation network based on semantic and multi-scale information; after training is completed, remove the semantic segmentation network and use the depth estimation network architecture containing only the depth decoder for derivation.

[0030] Step 4: Input the RGB image to be estimated and the millimeter-wave radar data into the depth estimation network obtained in step 3.2, and run it in derivation mode to realize the depth estimation of the scene.

[0031] Furthermore, the dual-branch initial feature extraction network includes an initial dense feature extraction module, an initial sparse feature extraction module, and a first fusion feature module;

[0032] The initial dense feature extraction module uses conventional convolution to extract RGB image features;

[0033] The initial sparse feature extraction module uses sparse convolution, which has an additional mask input for extracting features from millimeter-wave radar data.

[0034] The first fusion feature module receives the extracted image features and millimeter-wave radar data features, and splices and fuses them to obtain the first fusion feature output.

[0035] Furthermore, the structure of the initial dense feature extraction module is as follows:

[0036] The algorithm consists of 7×7 convolutional blocks, Batch Normalization (BN), ReLU, and 3×3 max pooling. Each group of 3×3 convolutional blocks has the following structure: 3×3 convolutional block - BN - ReLU - 3×3 convolutional block - BN. The 7×7 convolutional blocks have a stride of 2 and are used for initial image feature extraction. 3×3 max pooling is used to reduce the feature resolution. The three convolutional blocks serve as residual blocks, further refining the features to obtain a feature map with a resolution of 1 / 4 of the input.

[0037] The structure of the initial sparse feature extraction module is as follows:

[0038] The process involves 7×7 sparse convolution, 5×5 sparse convolution, 3×3 sparse convolution, 3×3 sparse convolution, 3×3 sparse convolution, and 3×3 max pooling, where the stride of the 7×7 sparse convolution is 2. The first 5 consecutive sparse convolutions are used to refine the features of the millimeter-wave radar data, and the 3×3 max pooling is used to reduce the feature resolution, resulting in a feature map with a resolution of 1 / 4 of the input.

[0039] The first fusion feature module includes a splicing layer and a 1×1 convolutional layer. The splicing layer receives the feature maps output by the initial dense feature extraction module and the initial sparse feature extraction module, splices them, and outputs them to the 1×1 convolutional layer. The 1×1 convolutional layer performs feature fusion on the spliced ​​features to obtain the first fusion feature.

[0040] Furthermore, the construction and training of the real-time depth estimation network based on semantic and multi-scale information all adopted the nuScenes dataset.

[0041] This invention provides a real-time depth estimation method based on semantic and multi-scale information.

[0042] On the one hand, semantic information is introduced into the depth estimation task without adding extra branches to the network architecture, thus maintaining the simplicity of the overall network architecture and enabling the network to run in real time at a high frame rate. During training, feature distillation is used to transfer semantic information to the depth estimation network, forcing the depth estimation network to learn semantic information during training. After training, i.e., during testing, semantically related branches are no longer retained, achieving the same performance as a multi-task architecture with only a single-branch structure. On the other hand, by exploring the role of multi-scale information in the depth estimation task, assuming that larger-scale features are suitable for estimating target edges and smaller-scale features are suitable for estimating large areas inside the target, a multi-scale fusion module is designed. The multi-scale fusion module maps decoder features of different scales to depth maps and confidence maps of corresponding scales, upsamples and reconstructs them, and then fuses them to obtain the final depth map, thus improving the depth estimation accuracy. It can be seen that the method adopted in this invention can achieve performance comparable to complex networks with a low-complexity single-stage architecture and obtain better estimation accuracy.

[0043] Compared with the prior art, the present invention has the following advantages:

[0044] 1) This invention achieves a root mean square error of 5.334m by combining multi-scale information and semantic information, which is the best accuracy among existing fusion methods.

[0045] 2) The network architecture proposed in this invention is highly simple, and the network runs in real time at a frame rate of 50 FPS. Attached Figure Description

[0046] Figure 1 This is a schematic diagram of a real-time depth estimation network architecture based on semantic and multi-scale information, as an example.

[0047] Figure 2 This is a schematic diagram of the dual-branch initial feature extraction network structure in an embodiment.

[0048] Figure 3 The flowchart of the multi-scale information fusion module in this embodiment is shown.

[0049] Figure 4 This is a schematic diagram of the depth estimation results for an example. Detailed Implementation

[0050] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0051] This embodiment provides a real-time depth estimation method based on semantic and multi-scale information, including the following steps:

[0052] Step 1: Construct a real-time depth estimation network based on semantic and multi-scale information.

[0053] like Figure 1 As shown, the real-time depth estimation network based on semantic and multi-scale information includes a dual-branch initial feature extraction network, a shared encoder, and a decoding network.

[0054] A dual-branch initial feature extraction network takes RGB image and millimeter-wave radar data as input, extracts features from each, and then fuses them to obtain the first fused feature output. For example... Figure 2 As shown, it includes an initial dense feature extraction module, an initial sparse feature extraction module, and a first fusion feature module.

[0055] The initial dense feature extraction module uses conventional convolution, and its structure is as follows:

[0056] The process consists of 7×7 convolutional blocks, batch normalization (BN), ReLU, and 3×3 max pooling. Each convolutional block is structured as follows: 3×3 convolutional block - batch normalization (BN) - ReLU - 3×3 convolutional block - batch normalization (BN). The stride of the 7×7 convolutional block is 2. Image data is first initially extracted using 7×7 convolutions, then 3×3 max pooling is used to reduce the feature resolution. The output features are then input into three consecutive residual blocks consisting of two convolutions with a 3×3 kernel size for further feature refinement.

[0057] F img =f(Maxpool(ReLU(BN(Conv(x))) img ))))) (1)

[0058] In equation (1), x img The input image is represented by Conv, the convolution operation by BN, the batch normalization layer by ReLU, the max pooling operation by Max pooling, and f(·) representing the residual block formed by convolutions. img This represents the output image features.

[0059] The initial sparse feature extraction module, used to extract millimeter-wave radar features, has the following structure:

[0060] The initial sparse feature extraction module uses a 7×7 sparse convolution followed by a 5×5 sparse convolution, then a 3×3 sparse convolution, followed by a 3×3 sparse convolution, and finally a 3×3 max pooling. The initial sparse feature extraction module has an additional mask input, which explicitly considers the sparsity of the input features. To reduce the computational cost of the network, the stride of the first sparse convolution is set to 2, and after refining the features using five consecutive sparse convolutions, max pooling is used to further reduce the feature resolution.

[0061] F radar =Maxpool(f spr (x radar (2)

[0062] In equation (2), x radar f represents a sparse depth map obtained from millimeter-wave radar. spr (·) represents a feature extraction module constructed from sparse convolutions, F radar This indicates the output millimeter-wave radar characteristics.

[0063] The first feature fusion module includes a stitching layer and a 1×1 convolutional layer. The stitching layer receives feature maps output from the initial dense feature extraction module and the initial sparse feature extraction module. After passing through the initial dense feature extraction module and the initial sparse feature extraction module, the feature resolution of both the image and the sparse depth is 1 / 4 of the input, so they can be directly stitched together for feature fusion. In feature fusion, this embodiment adopts a highly concise fusion method, that is, using a single 1×1 convolution to fuse the two types of features. Its output fused features are input to a shared encoder to obtain highly abstract high-level features.

[0064] F fus =BN(Conv[F img ,F radar (3)

[0065] In equation (3), [·,·] represents the splicing operation, F fus This indicates the fusion features of the output.

[0066] The shared encoder receives the first fused features and abstracts them to obtain the encoder feature output. The decoding network includes a semantic segmentation network and a depth estimation network. The semantic segmentation network architecture is the same as the depth decoder architecture, and its input is the encoder features output by the shared encoder. The semantic segmentation network is a pre-trained network structure used to provide semantic information about the scene to the depth estimation network during training.

[0067] The deep network consists of a deep decoder and a multi-scale information fusion module. The input to the deep decoder is the encoder features, which are used to decode the encoder features to obtain decoder features at different scales, which are then output to the multi-scale information fusion module. The multi-scale information fusion module processes these features according to the following steps to obtain the final depth map:

[0068] For features of different scales in the decoder, 1×1 convolutions are used to map them into depth maps and confidence maps of the corresponding scales. In order to achieve real-time computing, the mapping part uses a single convolution instead of a complex mapping structure.

[0069] The confidence map and depth map are upsampled to full resolution, and the depth estimation results obtained from features at different scales are fused pixel-wise based on the confidence.

[0070]

[0071] Where N = 4, C i (u,v) and D i (u,v) represent the confidence value and depth value of the i-th corresponding image at pixel location (u,v), respectively. D(u,v) represents the depth value at location (u,v) after aggregation based on the confidence values. It should be noted that the depth maps predicted by features at intermediate scales are also subject to supervision.

[0072] Step 2: Design the loss function for a real-time depth estimation network based on semantic and multi-scale information.

[0073] The loss function of the real-time depth estimation network based on semantic and multi-scale information consists of two parts: semantic information transfer-related loss and depth estimation-related loss.

[0074] The loss related to semantic information transfer is: the feature distillation loss generated by transferring semantic information to the deep estimation network using feature distillation, as shown in Equation (5):

[0075]

[0076] In equation (5), F s F represents the features of a semantic segmentation network. d The features of the deep estimation network are represented by two features with the same scale.

[0077] The depth estimation loss consists of three components: the error between the final depth map and the ground truth. As shown in equation (6); smoothing constraint As shown in equation (7); the error of the depth map in the multi-scale information fusion module As shown in equation (8);

[0078]

[0079] In equation (6), T is the set of all pixels in the image, m is the number of valid pixels, i.e., the total number of pixels with a true value greater than 0, where p i and y i These represent the predicted depth value and the actual depth value of pixel i, respectively.

[0080]

[0081] In equation (7), and This indicates taking the derivative with respect to the x and y directions;

[0082]

[0083] In equation (8), The loss term represents the loss for different intermediate depth maps, P = 4;

[0084] Therefore, the complete loss function of the designed real-time depth estimation network based on semantic and multi-scale information is shown in equation (9):

[0085]

[0086] In equation (9), ω1, ω2 and ω3 are hyperparameters, and ω1 = 0.01, ω2 = 0.5 and ω3 = 4;

[0087] Step 3: Train a real-time depth estimation network based on semantic and multi-scale information. The training process includes the following sub-steps:

[0088] 3.1. Train a semantic segmentation network based on the first fusion feature and the ground truth graph to obtain semantic segmentation information;

[0089] 3.2 Transfer the semantic segmentation information to the deep estimation network, and use the loss function obtained in step 2 for feedback to train the real-time deep estimation network based on semantic and multi-scale information;

[0090] Step 4: Input the RGB image to be estimated and the millimeter-wave radar data into the trained real-time depth estimation network based on semantic and multi-scale information, and run it in derivation mode to realize the depth estimation of the scene.

[0091] This embodiment uses the nuScenes dataset for both training and testing. The nuScenes dataset contains not only camera and LiDAR data but also millimeter-wave radar data, making it one of the few large datasets containing millimeter-wave radar data. Each scene in this dataset is 20 seconds long, with 40 keyframes, and each frame has a resolution of 1600×900. Furthermore, nuScenes includes driving scenes under various conditions, such as rain and nighttime, which increases the difficulty of depth estimation on this dataset. The invention uses PyTorch to deploy the network and trains it on a single NVIDIA GeForce GTX TITAN X. The images are resized from 1600×900 to 1600×704 for training. The batch size is set to 3, and the Adam optimizer with a learning rate of 0.0005 is used, with the learning rate halved every 5 batches. The results are as follows. Figure 4 As shown in the prediction depth map, the proposed algorithm can obtain a relatively accurate prediction depth value for each pixel, and the prediction of the target outline and small rod-shaped objects is also quite ideal.

[0092] To more clearly illustrate the effectiveness of the depth estimation method in this implementation, it is compared with existing technologies. The comparison results are as follows: Figure 1 As shown:

[0093] Table 1 Depth estimation results

[0094]

[0095]

[0096] As can be seen from Table 1, the method in this embodiment achieves the best performance compared to the existing schemes. Compared to the current best scheme RadarNet, the proposed network reduces RMSE by 0.555m and MAE by 0.359m.

Claims

1. A real-time depth estimation method based on semantic and multi-scale information, characterized in that, Includes the following steps: Step 1: Construct a real-time depth estimation network based on semantic and multi-scale information. Real-time depth estimation network based on semantic and multi-scale information, including a two-branch initial feature extraction network, a shared encoder and a decoding network; A dual-branch initial feature extraction network takes RGB image and millimeter-wave radar data as input, extracts features from them separately, and then fuses them to obtain the first fused feature output. The shared encoder receives the first fused feature and abstracts it to obtain the encoder feature output; The decoding network includes a semantic segmentation network and a depth estimation network; The semantic segmentation network is a pre-trained network structure used to provide semantic information of the scene for the depth estimation network during training. The deep network includes a deep decoder and a multi-scale information fusion module. The input of the deep decoder is the encoder features, which are used to decode the encoder features to obtain decoder feature outputs at different scales. The multi-scale information fusion module has a mapping structure and an upsampling unit. The mapping structure receives decoder features at different scales and maps them into depth maps and confidence maps of the corresponding scales. After reconstruction by the upsampling unit, they are fused to obtain the final depth map. Step 2: Design the loss function for a real-time depth estimation network based on semantic and multi-scale information. The loss function of the real-time depth estimation network based on semantic and multi-scale information consists of two parts: semantic information transfer-related loss and depth estimation-related loss. The semantic information transfer-related loss is: the feature distillation loss generated by transferring semantic information to the deep estimation network using feature distillation, as shown in Equation (5): （5） In equation (5), Features representing semantic segmentation networks This represents the characteristics of a deep estimation network. and Features of the same scale; The depth estimation loss consists of three components: the error between the final depth map and the ground truth. As shown in equation (6); smoothing constraint As shown in equation (7); the error of the depth map in the multi-scale information fusion module. As shown in equation (8); （6） In equation (6), Let m be the set of all pixels in the image. Middle truth value The total number of valid pixels with values ​​greater than 0, of which and These represent the predicted depth value and the actual depth value of pixel i, respectively. （7） In equation (7), and This indicates taking the derivative with respect to the x and y directions; （8） In equation (8), The loss term represents the loss for different intermediate depth maps, P=4; Therefore, the complete loss function of the designed real-time depth estimation network based on semantic and multi-scale information is shown in equation (9): （9） In equation (9), , and It is a hyperparameter, and , , ; Step 3: Train a real-time depth estimation network based on semantic and multi-scale information. The training process includes the following sub-steps: 3.

1. Train the semantic segmentation network based on the encoder features and truth map output by the shared coding network, obtain the semantic segmentation network weights, and fix their weights in subsequent training; 3.2 Transfer the semantic segmentation information to the depth estimation network, and use the loss function obtained in step 2 for feedback to train the real-time depth estimation network of semantic and multi-scale information; after training, remove the semantic segmentation network to obtain a depth estimation network architecture that only contains a depth decoder and a multi-scale information fusion module. Use this depth estimation network architecture that only contains a depth decoder for derivation. Step 4: Input the RGB image to be estimated and the millimeter-wave radar data into the depth estimation network obtained in step 3.2, and run it in derivation mode to realize the depth estimation of the scene.

2. The real-time depth estimation method based on semantic and multi-scale information according to claim 1, characterized in that: The dual-branch initial feature extraction network includes an initial dense feature extraction module, an initial sparse feature extraction module, and a first fusion feature module; The initial dense feature extraction module uses conventional convolution to extract RGB image features; The initial sparse feature extraction module uses sparse convolution, which has an additional mask input for extracting features from millimeter-wave radar data. The first fusion feature module receives the extracted image features and millimeter-wave radar data features, and splices and fuses them to obtain the first fusion feature output.

3. The real-time depth estimation method based on semantic and multi-scale information according to claim 2, characterized in that: The structure of the initial dense feature extraction module is as follows: The process consists of 7×7 convolutional blocks, Batch Normalization (BN), ReLU, and 3×3 max pooling. Each group comprises three consecutive 3×3 convolutional blocks, with the structure: 3×3 convolutional block - BN - ReLU - 3×3 convolutional block - BN. The 7×7 convolutional blocks have a stride of 2 and are used for initial image feature extraction. 3×3 max pooling is used to reduce the feature resolution. These three convolutional blocks serve as residual blocks, further refining the features to obtain a feature map with a resolution 1 / 4 of the input. The structure of the initial sparse feature extraction module is as follows: The process involves 7×7 sparse convolution, 5×5 sparse convolution, 3×3 sparse convolution, 3×3 sparse convolution, 3×3 sparse convolution, and 3×3 max pooling, where the stride of the 7×7 sparse convolution is 2. The first 5 consecutive sparse convolutions are used to refine the features of the millimeter-wave radar data, and the 3×3 max pooling is used to reduce the feature resolution, resulting in a feature map with a resolution of 1 / 4 of the input.

4. The real-time depth estimation method based on semantic and multi-scale information according to claim 2, characterized in that: The first fusion feature module includes a splicing layer and a 1×1 convolutional layer. The splicing layer receives the feature maps output by the initial dense feature extraction module and the initial sparse feature extraction module, splices them, and outputs them to the 1×1 convolutional layer. The 1×1 convolutional layer performs feature fusion on the spliced ​​features to obtain the first fusion feature.

5. A real-time depth estimation method based on semantic and multi-scale information according to any one of claims 1 to 4, characterized in that: The construction and training of the real-time depth estimation network based on semantic and multi-scale information all used the nuScenes dataset.

Images