Depth completion method for single-line lidar

Abstract

A depth completion method applied to a single-line laser radar, which processes the KITTI 64-line depth completion data set to obtain a single-line depth completion data set, and uses the single-line depth completion data set to train a dual-branch deep learning network, and then uses the RGB image and the aligned single-line depth map as the input of the neural network to obtain a dense depth map. The present invention replaces the 64-line laser radar with a single-line laser radar to achieve depth completion, significantly reducing the acquisition cost of the dense depth map. It can be applied not only to single-line laser radars, but also to any n-line (n≤64) laser radars, such as 4-line laser radars, 16-line laser radars, etc.

Description

Technical Field

[0001] The present invention relates to a technology in the field of image processing, and specifically to a depth completion method applied to a single-line laser radar. Background Art

[0002] In recent years, accurate and dense depth maps have become increasingly important in various tasks, such as augmented reality, drones, robot control, and autonomous driving. However, in outdoor environments, sensors such as near-infrared and structured light will fail due to strong interference and long distances in the outdoor environment. LiDAR has become the mainstream measurement method for outdoor environments due to its strong anti-interference ability and long measurement distance. However, the limited scanning lines of LiDAR will lead to high sparsity in the depth map it obtains. Taking the current Velodyne HDL-64E (64-line LiDAR) as an example, the effective value in the depth map it obtains is only about 4%. Such a sparse depth map is far from enough for practical applications. Therefore, how to generate accurate and dense depth maps based on the sparse depth maps of existing LiDARs has become a major research hotspot.

[0003] Existing technologies can only obtain sparse depth maps in outdoor environments. The current mainstream solutions use 64-line LiDAR and RGB cameras to obtain data, and then use the data of the two to generate dense depth maps, namely depth completion. 64-line LiDAR means that this type of LiDAR uses 64 scan lines. With the rise of deep learning in recent years, more and more teams have begun to try to use neural networks to achieve depth completion. Specifically, this type of solution uses a 64-line LiDAR and an RGB camera to obtain a 64-line sparse depth map and an RGB color image respectively, and align the two. Then the two are sent as input to the trained neural network to generate a dense depth map. Since the training data of the neural network is closely related to the detection accuracy, the existing training data sets such as KITTI use 64-line depth maps, which means that the system must use a 64-line LiDAR when in use, which severely limits the scope of application. Summary of the invention

[0004] In view of the above-mentioned deficiencies in the prior art, the present invention proposes a depth completion method applied to a single-line laser radar, in which a single-line laser radar is used to replace a 64-line laser radar to achieve depth completion, thereby significantly reducing the cost of acquiring a dense depth map. The method can be applied not only to a single-line laser radar, but also to any n-line (n≤64) laser radar, such as a 4-line laser radar, a 16-line laser radar, etc.

[0005] The present invention is achieved through the following technical solutions:

[0006] The present invention relates to a depth completion method applied to a single-line laser radar. A KITTI 64-line depth completion dataset is processed to obtain a single-line depth completion dataset. After a dual-branch deep learning network is trained using the single-line depth completion dataset, a dense depth map can be obtained by using an aligned single-line depth map and an RGB map as inputs of the neural network.

[0007] The processing refers to: converting a 64-line sparse depth map in an existing 64-line KITTI depth completion dataset into a single-line depth map, thereby obtaining a single-line depth completion dataset.

[0008] The dual-branch deep learning network includes a global branch and a local branch, wherein: the global branch uses the global semantic information in the RGB image to assist in predicting the boundary of the object to find areas with similar depths; the local branch focuses on the valid value area in the depth map and predicts the depth around the valid value area, while using the global features extracted by the global branch to correct the intrinsic errors of the sparse depth map.

[0009] The dual-branch deep learning network is trained by a combined loss function for single-line depth completion, where the combined loss function is L Final =L MSE +λL VN , where: λ is the weight parameter used to implement the mean square error loss function of pixel-level constraints Virtual normal error loss function for implementing 3D geometric constraints N p is the number of pixels with valid depth values ​​in the true depth map, and are the depth values ​​at position i in the predicted depth map and the true depth map respectively; N n is the number of randomly sampled virtual planes, and are the virtual normals of the i-th virtual plane in the predicted depth map and the true depth map, respectively.

[0010] The weight parameter λ is used to balance the two orders of magnitude, and is preferably 100.

[0011] Technical Effects

[0012] The present invention solves the problem that the existing depth completion technology can only be applied to 64-line laser radars but cannot be applied to single-line laser radars. The specific technical problems mainly include: 1. Lack of single-line depth completion data sets; 2. The depth information is extremely lacking in the single-line depth completion task, and it is difficult for the network of the prior art to generate an accurate dense depth map under this circumstance. For the first problem, the present invention obtains a single-line depth completion data set by processing the existing 64-line depth completion data set KITTI. For the second problem, the present invention uses a content-aware upsampling module proposed by the present invention in partial upsampling in the global branch while ensuring full use of scarce depth information by combining global branches with local branches. Different convolution kernels are used according to the semantic information of different positions, which can better utilize the semantic information of the RGB image to make up for the problem caused by the scarcity of depth information. At the same time, the pixel-level constraints and three-dimensional geometric constraints realized by the combined loss function proposed by the present invention also greatly improve the accuracy of the network of the present invention. With the above technology, the single-line depth completion error of the present invention reaches a very low level, with an absolute relative error of only 4.68%, which fully meets the needs of practical use. It can be applied to single-line laser radar to obtain a dense depth map, and with a slight transformation, it can be applied to any n-line (n≤64) laser radar, such as 4-line, 16-line, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 It is a schematic diagram of the system structure of the present invention;

[0014] Figure 2 Schematic diagram of RGB input in the embodiment;

[0015] Figure 3 A schematic diagram of single-line depth input in an embodiment;

[0016] Figure 4 Schematic diagram of network output effect of the embodiment. DETAILED DESCRIPTION

[0017] like Figure 1 As shown, this embodiment relates to a depth completion method applied to a single-line laser radar, and the specific steps include:

[0018] Step 1) Dataset generation: Convert the 64-line sparse depth map in the existing 64-line KITTI depth completion dataset into a single-line depth map to obtain a single-line depth completion dataset.

[0019] Each set of data in the 64-line depth completion dataset includes: an RGB image (input), an aligned 64-line sparse depth map (input), and a semi-dense depth map (true value, used as supervision for the neural network).

[0020] The conversion includes:

[0021] ①Convert the 64-line sparse depth map into a point cloud map;

[0022] ②Calculate the vertical angle of each 3D point in the point cloud

[0023] ③ Divide the vertical angle of each 3D point into 64 levels at intervals of 0.4°, so that each 3D point is assigned to the corresponding scan line;

[0024] ④Directly select the middle scan line to generate a single-line depth map.

[0025] In this embodiment, KITTI (64-line depth completion dataset) is used to generate a single-line depth completion dataset for single-line lidar, which is further divided into three non-overlapping training sets (85898 groups of data), validation sets (5852 groups of data) and test sets (1000 groups of data), where: the training set is used for training, the validation set is used to select the best network model, and the test set is used to evaluate the performance of the best network model. Each group of data contains an RGB image (input), an aligned single-line depth map (input), and a semi-dense depth map (true value).

[0026] Step 2) Arrange a neural network: Arrange a dual-branch deep learning network with a global branch (upper branch), a local branch (lower branch) and a normalization layer, wherein: the global branch uses the global semantic information in the RGB image to assist in predicting the boundary of the object to find areas with similar depths; the local branch focuses on the valid value area in the depth map and predicts the depth around the valid value area, while using the global features extracted by the global branch to correct the intrinsic errors of the sparse depth map.

[0027] The global branch includes two encoders A1 and A2 connected in parallel and a decoder A that uses the fused features of their outputs as input. The global branch relies on the global semantic information provided by the RGB image and can achieve better results in areas far away from the valid values ​​in the single-line depth map. The two encoders input the RGB image and the single-line depth map respectively and fuse the extracted features, and then the decoder A generates a global depth map, a global confidence map and global features.

[0028] The local branch includes: two sets of encoder-decoders (B and C) connected in series, and the local branch receives the fused global features and the single-line depth map to generate a local confidence map and a local depth map, wherein the global features guide the single-line depth, and in the area close to the valid value in the single-line depth map, the local branch can achieve better results.

[0029] The normalization layer normalizes the global confidence map and the local confidence map through the Softmax function to obtain a global weight map and a local weight map, which are respectively used to perform corresponding weighted summation on the subsequent global depth map and the local depth map to achieve fusion output.

[0030] The Softmax function Where: S i Represents the output of element i after the Softmax function, i represents element i, j represents all elements, that is, the Softmax value of the element, which is the ratio of the exponent of the element to the sum of the exponents of all elements. Here, element i refers to the global and local confidence maps respectively, and the sum of the exponents of all elements is the sum of the exponents of the global confidence map and the local confidence.

[0031] The fusion output is specifically: Where: D Final is the predicted depth map of the final output, C Global is the global confidence map, C Local is the local confidence map, D Global is the global depth map, D Local is the local depth map.

[0032] In this embodiment, the global branch uses two encoders to extract the features of the RGB image and the single-line depth map respectively and fuse them. Each encoder is composed of 13 residual blocks, and each residual block contains 2 groups of 3X1 and 1X3 convolutions. Then, the decoder is used to generate a global depth map, a global confidence map, and a global feature for guiding the local branch using the fused features. The decoder of the global branch uses a content-aware upsampling module and a transposed convolution to achieve upsampling, wherein the content-aware upsampling module is also a structure designed by the present invention for upsampling. The structure uses 3X3 convolution to dynamically generate different kernel matrices at different positions according to the content of the feature map. Compared with the transposed convolution, the same kernel matrix is ​​used for upsampling at all positions. This module designed by the present invention can better use the semantic information provided by the RGB image to complete the depth. The local branch uses global features and a single-line depth map as input, and a total of 2 sets of encoder-decoders are used. The encoder and decoder in the local branch only use ordinary convolution and transposed convolution. The local branch finally generates a local depth map and a local confidence map. Finally, the global depth map and the local depth map are fused with the confidence map to generate the final predicted depth map.

[0033] Step 3) Set the loss function: The dual-branch deep learning network obtained in step 2 is trained by a combined loss function for single-line depth completion, which consists of a mean square error loss function for implementing pixel-level constraints and a virtual normal error loss function for implementing 3D geometric constraints. Since the true depth map is semi-dense, only the valid value part of the true depth map is used to calculate the loss.

[0034] The virtual normal error loss function randomly samples several groups of points on the predicted depth map and the true depth map to form a virtual plane, each group of points contains 3 non-collinear points, and then generates virtual normals based on several groups of virtual planes. Finally, the loss is calculated using the virtual normals of the predicted depth map and the virtual normals of the true depth map.

[0035] Preferably, in order to make the mean square error loss function and the virtual normal error loss function at the same order of magnitude, the method sets the weight of the virtual normal loss to 100 and the weight of the mean square error loss to 1, so that the network focuses on pixel-level prediction while also focusing on three-dimensional geometric information.

[0036] A neural network that can be applied to single-line lidar is trained through the above data set. When in use, only the RGB image and the aligned single-line depth map need to be input to directly obtain an accurate and dense depth map. The single-line lidar and RGB camera used are both low-cost.

[0037] In this embodiment, a global depth map, a local depth map, and a final predicted depth map are obtained at each training iteration, and the above-mentioned combined loss function is used to calculate the loss of these three and the true value, and the AdamW optimizer is used to perform gradient descent training on the network according to the loss. Since the global depth and local depth are only used as intermediate supervision, the weight of the loss is set to 0.1, and the weight of the final predicted depth loss is 1.

[0038] Preferably, during the process of training the network, the generalization performance of the network is evaluated using a validation set, and the best set of network parameters is stored as the parameters of the final network.

[0039] This embodiment will finally use this set of parameters as the parameters of the final network model, and use the test set to evaluate its performance. A total of 4 sets of indicators are used to evaluate the network, as shown in Table 1. Since the true depth map is semi-dense, only the valid value part of the true depth map is used to calculate the error.

[0040] Table 1 Performance of the network of the present invention on the test set

[0041] network Root mean square error Mean absolute error Square relative error Absolute relative error Ours 2921.94mm 1080.99mm 0.97％ 4.68％

[0042] It can be found that the network of the present invention has achieved good results in various indicators, among which the absolute relative error is only 4.68%, which means that if the true depth value is 1m, the error of the completed depth value is only ±4.68cm, which is enough for practical applications.

[0043] like Figure 4 As shown, this is the performance of the network of the present invention on the single-line depth completion test set. It can be found that the network designed by the present invention takes the RGB image and the aligned single-line depth image as input, and can accurately generate a dense depth map, in which the boundaries of the car and the cyclist are accurately generated.

[0044] Table 2 Comparison between the present invention and existing advanced depth completion methods

[0045] network Root mean square error Mean absolute error Square relative error Absolute relative error Ours 2921.94mm 1080.99mm 0.97％ 4.68％ Revisiting 3356.27mm 1374.82mm 1.53％ 6.88％ NConv-CNN 6607.35mm 3035.96mm 7.45％ 16.33％

[0046] Existing depth completion networks are all designed based on 64-line laser radars. Although these networks can be used for single-line laser radars by replacing the 64-line depth map during training with the single-line depth map generated by the present invention, they are not optimized for the task of single-line laser radar, so the effect is often poor. Table 2 compares some advanced methods in the current 64-line depth completion task. When the single-line depth completion data set generated by the present invention is used for training, it can be found that the network designed by the present invention has great advantages in various indicators. In terms of the main indicator of root mean square error, the error of the network of the present invention is reduced by more than 12.94% compared with those advanced networks used for 64-line depth completion.

[0047] Table 3 shows the ablation experiment of the content-aware upsampling module and the combined loss function in the present invention.

[0048] network Root mean square error Mean absolute error Square relative error Absolute relative error Ours(Full) 2921.94mm 1080.99mm 0.97％ 4.68％ w / o CAUM 3013.59mm 1128.84mm 1.01％ 4.82％ w / o VNL 3098.18mm 1208.70mm 1.34％ 5.68％

[0049] This is an ablation experiment of the content-aware upsampling module and the combined loss function in the present invention, where ours(full) represents the complete structure used in the present invention, w / o CAUM (without Content-Aware Upsampling Module) represents replacing the content-aware upsampling module with a transposed convolution, and w / o VNL (without VirtualNormal Loss) represents replacing the combined loss function used in the present invention with a mean square error loss function, that is, removing the virtual normal loss. The experimental results show that the content-aware upsampling module and virtual normal loss designed by the present invention can greatly improve the accuracy of the network of the present invention in the single-line depth completion task.

[0050] After the trained network model is obtained in this embodiment, the present invention deploys the model in an actual scenario, thereby obtaining an accurate and dense depth map at a low cost. For the deployment of the RGB camera and the single-line lidar, it is necessary to ensure that the two are deployed on the same platform in a relatively fixed position and align the two at the same time. After that, the network model needs to be deployed in the hardware platform. Finally, during the operation, the aligned RGB image and the single-line depth map obtained in real time are simultaneously fed into the network, so that an accurate and dense depth map can be obtained at a very low cost.

[0051] After specific actual experiments, the present invention uses the single-line depth completion test set generated by KITTI to simulate the actual operating environment, and uses the aligned single-line depth map and RGB map as network input to simulate the deployment of RGB cameras and single-line lidars in real scenes on the same platform. The test set contains a total of 1000 sets of data, which is equivalent to completing the single-line depth completion of 1000 frames of images in the simulated scene. The quantitative error of 1000 frames is shown in Table 1. The qualitative results of this invention select 1 set of data for display.

[0052] The existing technology can only generate dense depth maps based on 64-line laser radars because the sparse depth maps of the depth completion data set are limited to 64 lines. Moreover, even when the network of the existing technology is trained using the data set generated by the present invention, the accuracy is far inferior to that of the network designed by the present invention because it has not been optimized for the single-line depth completion task. At the same time, when trained using the single-line depth completion data set generated by the present invention, the error of the network designed by the present invention is reduced by more than 12.94% in terms of the main indicator of root mean square error compared to those advanced networks used for 64-line depth completion.

[0053] The above-mentioned specific implementation can be partially adjusted in different ways by those skilled in the art without departing from the principle and purpose of the present invention. The protection scope of the present invention shall be based on the claims and shall not be limited by the above-mentioned specific implementation. Each implementation scheme within its scope shall be subject to the constraints of the present invention.

Claims

1. A depth completion method applied to a single-line laser radar, characterized in that: The KITTI 64-line depth completion dataset is processed to obtain a single-line depth completion dataset. After using the single-line depth completion dataset to train a dual-branch deep learning network, a dense depth map can be obtained by using the aligned single-line depth map and RGB map as the input of the neural network. The processing refers to: converting a 64-line sparse depth map in an existing 64-line KITTI depth completion dataset into a single-line depth map, thereby obtaining a single-line depth completion dataset; The dual-branch deep learning network includes a global branch and a local branch, wherein: the global branch uses the global semantic information in the RGB image to assist in predicting the boundary of the object to find the area with similar depth; the local branch focuses on the valid value area in the depth map and predicts the depth around the valid value area, while using the global features extracted by the global branch to correct the inherent error of the sparse depth map; The dual-branch deep learning network is trained by a combined loss function for single-line depth completion, which is composed of a mean square error loss function for implementing pixel-level constraints and a virtual normal error loss function for implementing three-dimensional geometric constraints.

2. The depth completion method applied to a single-line laser radar according to claim 1, characterized in that: The combined loss function is L Final =L MSE +λL VN , where: λ is the weight parameter used to implement the mean square error loss function of pixel-level constraints Virtual normal error loss function for implementing 3D geometric constraints N p is the number of pixels with valid depth values ​​in the true depth map, and are the depth values ​​at position i in the predicted depth map and the true depth map respectively; N n is the number of randomly sampled virtual planes, and are the virtual normals of the i-th virtual plane in the predicted depth map and the true depth map, respectively.

3. The depth completion method applied to a single-line laser radar according to claim 2, characterized in that: The weight parameter λ is used to balance the magnitudes of the two and is set to 100.

4. The depth completion method applied to a single-line laser radar according to claim 1, characterized in that: Each set of data in the 64-line depth completion dataset includes: an RGB image, an aligned 64-line sparse depth map, and a semi-dense depth map; The conversion includes: ①Convert the 64-line sparse depth map into a point cloud map; ②Calculate the vertical angle of each 3D point in the point cloud ③ Divide the vertical angle of each 3D point into 64 levels at intervals of 0.4°, so that each 3D point is assigned to the corresponding scan line; ④Directly select the middle scan line to generate a single-line depth map.

5. The depth completion method applied to a single-line laser radar according to claim 1, characterized in that: The dual-branch deep learning network specifically includes: a dual-branch deep learning network of a global branch, a local branch and a normalization layer, wherein: the global branch uses the global semantic information in the RGB image to assist in predicting the boundaries of objects to find areas with similar depths; the local branch focuses on the valid value area in the depth map and predicts the depth around the valid value area, while using the global features extracted by the global branch to correct the intrinsic errors of the sparse depth map.

6. The depth completion method for single-line laser radar according to claim 5, characterized in that: The global branch includes two encoders in parallel and a decoder that uses the features of their output fusion as input. The global branch relies on the global semantic information provided by the RGB image and can achieve better results in areas far away from the valid values ​​in the single-line depth map. The two encoders input the RGB image and the single-line depth map respectively and fuse the extracted features, and then the decoder generates a global depth map, a global confidence map and global features.

7. The depth completion method applied to a single-line laser radar according to claim 5, characterized in that: The local branch includes: two sets of encoder-decoders connected in series, and the local branch receives the fused global features and the single-line depth map to generate a local confidence map and a local depth map, wherein the global features play a role in guiding the single-line depth. In the area close to the valid value in the single-line depth map, the local branch can achieve better results.

8. The depth completion method applied to a single-line laser radar according to claim 5, characterized in that: The normalization layer normalizes the global confidence map and the local confidence map through the Softmax function to obtain a global weight map and a local weight map, which are respectively used to perform corresponding weighted summation on the subsequent global depth map and the local depth map to achieve fusion output.

9. The depth completion method for single-line laser radar according to claim 8, characterized in that: The Softmax function Where: S i Represents the output of element i after the Softmax function, i represents element i, j represents all elements, that is, the Softmax value of the element, which is the ratio of the exponent of the element to the sum of the exponents of all elements. The element i here refers to the global and local confidence maps respectively, and the sum of the exponents of all elements is the sum of the exponents of the global confidence map and the local confidence.

10. The depth completion method applied to a single-line laser radar according to claim 5, characterized in that: The fusion output is specifically: Where: D Final is the predicted depth map of the final output, C Global is the global confidence map, C Local is the local confidence map, D Global is the global depth map, D Local is the local depth map.

Images