Semantic information fused laser radar and vision fusion depth estimation system and method

Abstract

The invention belongs to the technical field of automatic driving, and particularly relates to a semantic information fused laser radar and vision fusion depth estimation system and method, the system comprises a first-stage network and a second-stage network, in the first-stage network, RGB image features and radar point cloud features are fused through a double encoder-single decoder structure, and a sparse depth map is generated; in the second-level network, semantic information obtained through RGB image segmentation is introduced to realize hierarchical constraint, channel feature self-adaption and an attention mechanism are matched to process the sparse depth map, a dense depth map is obtained, and a final depth map is generated; in the process, a multi-constraint loss function composed of LiDAR branch loss, sparse depth map loss and final prediction result loss is designed, and the system is optimized so as to guide the network to carry out depth estimation more accurately. Compared with the prior art, a more comprehensive and accurate depth estimation result is obtained.

Description

Technical Field

[0001] The present invention belongs to the technical field of autonomous driving, and in particular relates to a laser radar and vision fusion depth estimation system and method that integrates semantic information. Background Art

[0002] Depth estimation technology is a key technology in modern computer technology. It enables autonomous systems to understand and perceive three-dimensional spatial information, thereby performing precise interactions and operations in complex environments. This technology is widely used in fields such as augmented reality (AR), robotic navigation, and autonomous driving.

[0003] At present, the main methods for implementing depth estimation can be simply divided into two categories:

[0004] The first method is to directly measure the depth data of the scene through hardware devices. Commonly used hardware devices include active sensors such as radar and LiDAR. These sensors usually emit laser beams or electromagnetic waves and obtain detailed depth point cloud data based on the reflected signals on the surfaces of surrounding objects, thereby constructing accurate three-dimensional surface models and depths. However, these devices are usually limited to specific application scenarios and are relatively expensive.

[0005] The second method is to recover the depth information of the environment directly from a single image through monocular depth estimation technology. By analyzing the features and patterns in the image, the depth estimation algorithm can estimate the depth estimation model of the distance from each pixel corresponding to the object point to the camera from the RGB image. Its advantage is that it does not rely on expensive hardware equipment and has a wide range of application potential. However, in some practical application scenarios, it is difficult to obtain a large amount of real depth data as training samples; at the same time, the performance will be significantly affected in insufficient lighting conditions (such as at night or indoor environments) or bad weather (such as heavy rain, dense fog or snowstorms).

[0006] The two common depth estimation methods mentioned above have their own advantages and limitations. In recent years, researchers have begun to try new methods that combine hardware measurement with monocular depth estimation algorithms.

[0007] In the application scenarios of autonomous driving, the road conditions and scene structures are relatively fixed. During driving, the vehicle mainly relies on the depth information of the target area, such as lane lines, surrounding vehicles and pedestrians, etc., while the reliance on the depth information of non-target areas such as the sky, buildings and vegetation is relatively low. Existing applications mainly focus on improving the overall performance and reducing the error of depth estimation by improving the network structure, without fully considering the particularity of autonomous driving scenarios and the difference between actual application needs and theoretical research.

[0008] Therefore, it is necessary to study a depth estimation method that fully considers the particularities of autonomous driving scenarios and the differences between actual application requirements and theoretical research, so as to obtain more comprehensive, accurate and precise depth estimation results, so that the autonomous driving system can more accurately understand the three-dimensional structure of the surrounding environment, thereby improving the reliability of the autonomous driving system. Summary of the invention

[0009] The purpose of the present invention is to provide a LiDAR and vision fusion depth estimation system and method that integrates semantic information, fully considering the particularity of autonomous driving scenarios and actual application needs, using LiDAR and visible light camera data, combining deep learning technology with hierarchical constraint technology, and using semantic segmentation to integrate semantic information into network training, so as to obtain a more comprehensive, accurate and deep estimation result.

[0010] To achieve the above object, the present invention adopts the following technical solution:

[0011] A laser radar and visual depth fusion estimation system integrating semantic information includes a first-level network and a second-level network:

[0012] The first-level network includes a first encoder and a first decoder, wherein the first encoder is a dual encoder including an image branch network and a radar point cloud branch network; the image branch network is used to extract image features from RGB image data; the radar point cloud branch network includes a sparse pre-mapping module and a residual block connected after the sparse pre-mapping module, the sparse pre-mapping module is used to extract a first point cloud feature from the radar point cloud data, and the residual block is used to extract a second point cloud feature, i.e., a final point cloud feature, from the first point cloud feature; the first decoder is used to receive the image features provided by the image branch network and the point cloud features provided by the radar branch network, and fuse the image features and the point cloud features to generate a sparse depth map;

[0013] The second-level network includes a semantic segmentation module, a second encoder and a second decoder; the semantic segmentation module is used to perform semantic segmentation on the RGB image data to obtain semantic information; the second encoder receives the sparse depth map and the semantic segmentation information, and obtains a fusion feature by splicing the sparse depth map and the semantic segmentation information; the second decoder receives the fusion feature and decodes it to generate a dense depth map, and generates a final depth prediction map based on the dense depth map.

[0014] Furthermore, the image branch network is a residual network RseNet34.

[0015] Furthermore, the sparse pre-mapping module includes five sequentially connected SPM structures, whose input is radar point cloud data and an initial binary observation mask, and whose output is a first point cloud feature; the initial binary observation mask is generated according to the radar point cloud data, and the subsequent binary observation mask is determined by performing a maximum pooling operation on the output of the previous SPM structure; each SPM structure includes five sequentially connected sparse invariant convolutional layers, whose convolution kernels are 7×7, 5×5, 3×3, 3×3, and 1×1, respectively, the number of channels of the first four is 16, and the number of channels of the last one is 1.

[0016] Furthermore, the first decoder includes four up-projection (UpProj) modules connected in sequence.

[0017] Furthermore, the second encoder includes a feature adaptation module and a multi-layer fusion module. The feature adaptation module input is the sparse depth map output by the first decoder, which is used to extract features from the sparse depth map, and the extracted features are output to the first input end of the multi-fusion module; the second input end of the multi-fusion module receives semantic information provided by the semantic segmentation module, and uses the channel attention mechanism to weightedly fuse the semantic information and the features extracted by the feature adaptation module to obtain fused features.

[0018] Furthermore, the second decoder includes four up-projection UpProj modules and a bilinear interpolation layer connected after the four up-projection UpProj modules.

[0019] Furthermore, the loss function used in the optimization of the above-mentioned laser radar and vision fusion depth estimation system that integrates semantic information is:

[0020] Loss = λ1L LiDAR +λ2L Mid +λ3L Final

[0021] Among them, λ1, λ2, λ3 are hyperparameters λ1L LIDAR is the loss function of the radar branch network; λ2L Mid represents the loss of the first-level network; λ3LFinal Represents the loss of the final prediction result.

[0022] Furthermore, the semantic segmentation module uses a pre-trained eepLabV3+ model to complete semantic segmentation.

[0023] A depth estimation method of laser radar and vision fusion by fusing semantic information, comprising the following steps:

[0024] Step 1: Use the residual network to extract image features from RGB image data;

[0025] Step 2: extracting point cloud features from radar point cloud data using a laser radar branch network, wherein the laser radar branch network includes a sparse pre-mapping module and a residual block connected to the sparse pre-mapping module, wherein the sparse pre-mapping module is used to extract a first point cloud feature from the radar point cloud data, and the residual block is used to extract a second point cloud feature, i.e., a final point cloud feature, from the first point cloud feature;

[0026] Step 3: Use the semantic segmentation module to perform semantic segmentation on the RGB image data to obtain semantic segmentation information;

[0027] There is no order of precedence for steps 1 to 3;

[0028] Step 4: Perform cross-modal fusion of image features and point cloud features to obtain fusion features;

[0029] Step 5: Based on the fused features, a sparse depth map is generated using the first decoder;

[0030] Step 6: Use the second encoder to concatenate the semantic segmentation information and the sparse depth map, and then input them into the second decoder to obtain a dense depth map through decoding;

[0031] Step 7: Generate the final depth prediction map based on the dense depth map.

[0032] Furthermore, the step 7 also includes visualization processing of the final depth prediction map.

[0033] The present invention provides a LiDAR and visual depth fusion estimation system and method that integrates semantic information. In the first-level network, RGB image features and radar point cloud features are integrated through a dual encoder-single decoder structure to generate a sparse depth map. In the second-level network, the semantic information obtained by RGB image segmentation is introduced to realize hierarchical constraints, and the sparse depth map is processed with channel feature adaptation and attention mechanism to obtain a dense depth map to generate the final depth map. In this process, a multi-constraint loss function composed of LiDAR branch loss, sparse depth map loss and final prediction result loss is designed to optimize the system so as to guide the network to perform depth estimation more accurately. Compared with the prior art, the present invention obtains more comprehensive, accurate and high-quality depth estimation results. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 It is a system architecture of laser radar and visual depth fusion estimation that fuses semantic information in an embodiment;

[0035] Figure 2 is a structural diagram of RseNet34 adopted by the image branch network of the embodiment;

[0036] Figure 3 is a schematic diagram of the structure of a sparse pre-mapping module (SPM) in a radar branch network of an embodiment;

[0037] Figure 4 Schematic diagram of sparse invariant convolution in the sparse pre-mapping module in the embodiment;

[0038] Figure 5 is a schematic diagram of the UpProj module in the decoder of the first-level network of the embodiment;

[0039] Figure 6 is a schematic diagram of the structure of an encoder in the second-level network of an embodiment;

[0040] Figure 7 It is a visualization processing analysis diagram of the final depth map schematic diagram of the embodiment, where a is the RGB input; b is the output of the unipolar network, and c is the output after introducing semantic segmentation information. DETAILED DESCRIPTION

[0041] The technical solution of the present invention is described in detail below in conjunction with the accompanying drawings and embodiments.

[0042] like Figure 1 As shown, this embodiment provides a laser radar and visual depth fusion estimation system that integrates semantic information, including a first-level network and a second-level network.

[0043] The first-level network includes a first encoder and a first decoder. The first encoder includes an image branch network and a radar point cloud branch network.

[0044] The image branch network is used to extract image features from RGB image data. It aims to process RGB images through this structure to provide scene information for depth estimation and to support the conversion of depth information from the original sparse LiDAR point cloud to dense depth information. Figure 2 As shown, the image branch network of this embodiment adopts the residual network RseNet34, which has two training paths, the main path is a traditional convolutional layer stack, and the side branch is a jump connection. The main branch is added through a series of convolutional layers, side branches and the main branch after operation, and then an excitation function is performed, so that the network can learn the residual between input and output instead of directly learning the output, thereby alleviating the gradient vanishing problem in deep networks and enabling the network to successfully train deeper structures.

[0045] The radar point cloud branch network consists of a sparse pre-mapping module and a residual block connected to the sparse pre-mapping module; the sparse pre-mapping module is used to extract the first point cloud feature from the radar point cloud data; the residual block is used to extract the second point cloud feature, that is, the final point cloud feature, from the first point cloud feature.

[0046] The formula for a common convolution operation is as follows:

[0047]

[0048] The convolution kernel size is 2k+1, the weight is w, and the bias is b. The above standard convolution operation is effective when the input data is complete, non-sparse and continuous. However, when the input data is sparse radar point cloud data, the traditional convolution operation cannot effectively process this incomplete information, which will destroy the discreteness of the original data and turn it into continuous data.

[0049] The sparse invariant convolution explicitly takes sparsity into account by calculating the effective pixels and properly normalizing the output. The calculation formula is as follows:

[0050]

[0051] Where o is the data mask. When the input matrix x u,v Valid time u,v is 1, when the input matrix x u,v When invalid, o u,v is 0; a small ∈ is added to the denominator to avoid u+i,v+i The filter position is divided by zero when it is observed. At the same time, the sparse invariant convolution determines the subsequent observation mask by utilizing the maximum pooling operation to keep track of the input valid information state and make it available to the next layer of the network.

[0052] Therefore, the sparse pre-mapping module in this embodiment is composed of five SPM structures connected in sequence, whose input is radar point cloud data and initial binary observation mask, and output is the first point cloud feature; the initial binary observation mask is generated according to the radar point cloud data, and the subsequent binary observation mask is determined by performing the maximum pooling operation on the output of the previous SPM structure; Figure 3-Figure 4 As shown in the figure, each SPM structure consists of five sparse invariant convolutional layers connected in sequence, whose convolution kernels are 7×7, 5×5, 3×3, 3×3, and 1×1 respectively. The number of channels of the first four is 16, and the number of channels of the last one is 1.

[0053] The first decoder receives the image features provided by the image branch network and the point cloud features provided by the radar branch network, and fuses the image features and the point cloud features to generate a sparse depth map. The first decoder of this embodiment is composed of four up-projection (UpProj) modules connected in sequence. Figure 5 As shown, the structure of each up-projection (UpProj) module is as follows.

[0054] First, the input information of each is the feature F0 with a resolution of W*H. It is first passed through the inverse pooling layer composed of a 2×2 convolution kernel to double the size of the feature map, and a 2W*2H feature map F1 is obtained. Then F1 is further processed by the convolution layer and the nonlinear activation function ReLU, and the skip connection is used to convert Figure 5 In the skip connection, after passing F1 through convolution kernels of size 5 and 3 respectively, F1 will be passed through an additional convolution kernel of size 5, and the two different feature maps calculated by F1 will be directly added to obtain the new feature F2, which is the output of the residual module in the figure. Finally, F2 with a resolution of 2W*2H is activated by the ReLU function to increase the nonlinearity of the network, and F2 is mapped to a feature map of size W*H using a 3×3 convolution, and finally the feature map is restored to 2W*2H using bilinear upsampling.

[0055] The second-level network includes a semantic segmentation module, a second encoder, and a second decoder. The semantic segmentation module obtains semantic segmentation information in the driving scene by semantically segmenting the RGB image data. The semantic segmentation module uses existing semantic segmentation technology, such as the DeepLabV3+ model, to semantically segment the RGB image, and the segmentation process includes:

[0056] 1) The semantic information in the application scenario is divided into importance levels according to the needs. The purpose of the level division is to determine the target area and the non-target area: The importance level division of the semantic information in the driving scenario in this embodiment is shown in Table 1, where importance level 0 is used as the non-target area, and importance levels 1 and 2 are used as the target area.

[0057] Table 1 Classification of importance of semantic information

[0058]

[0059] 2) Based on the importance level classification results, a category label is assigned to each pixel in the RGB image to obtain the semantic segmentation result and realize the semantic understanding of each pixel in the RGB image.

[0060] In the embodiment, if the semantic segmentation result is used directly, the boundary area will generate jagged noise, and the use of surrounding information is easily ignored in the pixel-level classification. To solve this problem, enhance the boundary between the target and the non-target, and improve the accuracy of the target area prediction, this embodiment processes the output of the semantic segmentation by using a sliding window and pooling operation.

[0061] The second encoder receives the sparse depth map D and the semantic segmentation information S, and obtains the fusion feature F by splicing D and S at the channel scale. Its network structure includes a feature adaptation module and a multi-layer fusion module. The input of the feature adaptation module is the sparse depth map with residual connection output by the first decoder, which is used to extract features from the sparse depth map to obtain the feature F. d ; The first input of the multi-fusion module is F d , the second input is S, and the channel attention mechanism is used to fuse the features extracted from the sparse depth map with the semantic information to obtain F. The specific operation is as follows Figure 6 As shown:

[0062] The feature adaptation module refers to the residual block in the ResNet network. D is first convolved by a convolution layer with a convolution kernel of 1×1, and then processed in two ways. One way is used for the feature refinement mapping module to obtain the feature f1(Conv(D)), and the other way is added to f1(Conv(D)) after the convolution operation and then output. The formula is:

[0063] Yi=ReLU(Conv(S)+f1(Conv(D)))

[0064] Among them, Fi and Yi represent the input and output of the i-th layer, Conv represents the convolution operation, and f1 represents the feature refinement mapping operation.

[0065] The specific operations of the multi-fusion module are as follows:

[0066] (1) Concatenate the semantic information and the decoder features processed by the feature adaptation module at the channel scale.

[0067] (2) The spliced ​​results are globally pooled and then go through a convolution layer with a convolution kernel size of 1*1,

[0068] Relu activation function, convolution layer with a convolution kernel size of 1*1, and Sigmoid activation function, the output is the weight.

[0069] (3) The final output is:

[0070] y=σ(f2(Global[Yi,FSi])⊙Yi+Fsi

[0071] Among them, Global represents the global pooling operation, σ represents the Sigmoid activation function, f2 represents the specific convolution operation, ⊙ represents the pixel-level multiplication operator for the multiplication of two matrices, FSi represents the encoder feature, and Yi represents the processed decoder output.

[0072] The input of the second decoder is the fused features, and the output is the predicted depth map. The second decoder first decodes the fused features to generate a dense depth map, and then generates a final depth prediction map based on the dense depth map. The structure of the second decoder in this embodiment is the same as that of the first decoder, and will not be repeated here.

[0073] The loss function used in the depth estimation system of the laser radar and vision fusion fusion of semantic information in this embodiment is optimized as follows:

[0074] Loss = λ1L LiDAR +λ2L Mid +λ3L Final

[0075] Among them, λ1, λ2, λ3 are hyperparameters λ1L LIDAR is the loss function of the radar branch network; λ2L Mid represents the loss of the first-level network; λ3L Final Represents the loss of the final prediction result.

[0076] A depth estimation method of laser radar and vision fusion by fusing semantic information, comprising the following steps:

[0077] Step 1: Use the residual network to extract image features from RGB image data;

[0078] Step 2: Use the LiDAR branch network to extract point cloud features from the radar point cloud data;

[0079] The sparse pre-mapping module (SPM) and the residual block extract features from the point cloud data to obtain point cloud features;

[0080] Step 3: Use the semantic segmentation network to semantically segment the RGB image data to obtain semantic segmentation information;

[0081] There is no order of precedence for steps 1 to 3;

[0082] Step 4: Perform cross-modal fusion of image features and point cloud features to obtain fusion features;

[0083] Step 5: Based on the fused features, a sparse depth map is generated using the first decoder;

[0084] Step 6: After the semantic segmentation information and the sparse depth map are spliced ​​at the channel scale, they are input into the codec to obtain a dense depth map through codec processing;

[0085] Step 7: Generate a final depth prediction map based on the dense depth map, and visualize the final depth prediction map. Figure 7 : is a visualization analysis diagram of the final depth map diagram of the embodiment, where a is the RGB input; b is the output of the unipolar network, and c is the output after the semantic segmentation information is introduced. Figure 7 It can be seen that compared with the original single-stage network, the lidar and vision fusion depth estimation system and method of this embodiment that integrates semantic information can restore more accurate detail information in the selected area, making the distinction between the target object and the background more obvious, and at the same time, the depth prediction of the edge of the target area and the non-target area is more accurate.

[0086] It is to be understood that the present invention is described by some embodiments, and it is known to those skilled in the art that various changes or equivalent substitutions may be made to these features and embodiments without departing from the spirit and scope of the present invention. In addition, under the teachings of the present invention, these features and embodiments may be modified to adapt to specific circumstances and materials without departing from the spirit and scope of the present invention. Therefore, the present invention is not limited by the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are within the scope of protection of the present invention.

Claims

1. A laser radar and visual depth fusion estimation system that integrates semantic information, comprising a first-level network and a second-level network, characterized in that: The first-level network includes a first encoder and a first decoder, wherein the first encoder is a dual encoder including an image branch network and a radar point cloud branch network; the image branch network is used to extract image features from RGB image data; The radar point cloud branch network includes a sparse pre-mapping module and a residual block connected to the sparse pre-mapping module, wherein the sparse pre-mapping module is used to extract a first point cloud feature from the radar point cloud data, and the residual block is used to extract a second point cloud feature, i.e., a final point cloud feature, from the first point cloud feature; a first decoder is used to receive image features provided by the image branch network and point cloud features provided by the radar branch network, and fuse the image features and the point cloud features to generate a sparse depth map; The second-level network includes a semantic segmentation module, a second encoder and a second decoder; the semantic segmentation module is used to perform semantic segmentation on the RGB image data to obtain semantic information; the second encoder receives the sparse depth map and the semantic segmentation information, and obtains a fusion feature by splicing the sparse depth map and the semantic segmentation information; The second decoder receives the fused features and decodes them to generate a dense depth map, and generates a final depth prediction map according to the dense depth map.

2. The laser radar and visual depth fusion estimation system integrating semantic information according to claim 1, characterized in that: The image branch network is the residual network RseNet34.

3. The laser radar and visual depth fusion estimation system integrating semantic information according to claim 2, characterized in that: The sparse pre-mapping module includes five SPM structures connected in sequence, whose input is radar point cloud data and an initial binary observation mask, and whose output is a first point cloud feature; The initial binary observation mask is generated according to the radar point cloud data, and the subsequent binary observation mask is determined by performing a maximum pooling operation on the output of the previous SPM structure; each SPM structure includes five sparse invariant convolutional layers connected in sequence, and the convolution kernels are 7×7, 5×5, 3×3, 3×3, and 1×1, respectively. The number of channels of the first four is 16, and the number of channels of the last one is 1.

4. The laser radar and visual depth fusion estimation system integrating semantic information according to claim 3, characterized in that: The first decoder includes four upper projection modules connected in sequence.

5. The laser radar and visual depth fusion estimation system integrating semantic information according to claim 4, characterized in that: The second encoder includes a feature adaptation module and a multi-layer fusion module. The feature adaptation module input is the sparse depth map output by the first decoder, which is used to extract features from the sparse depth map, and the extracted features are output to the first input end of the multi-fusion module; the second input end of the multi-fusion module receives the semantic information provided by the semantic segmentation module, and uses the channel attention mechanism to weightedly fuse the semantic information and the features extracted by the feature adaptation module to obtain a fused feature.

6. The laser radar and visual depth fusion estimation system integrating semantic information according to claim 5, characterized in that: The second decoder includes four up-projection modules and a bilinear interpolation layer connected after the four up-projection modules.

7. The laser radar and visual depth fusion estimation system integrating semantic information according to any one of claims 1 to 6, characterized in that: The loss function used in the above-mentioned depth estimation system of laser radar and vision fusion that integrates semantic information is as follows: Loss=λ1L LiDAR +λ2L Mid +λ3L Final Among them, λ1, λ2, λ3 are hyperparameters λ1L LIDAR is the loss function of the radar branch network; λ2L Mid represents the loss of the first-level network; λ3L Final Represents the loss of the final prediction result.

8. The laser radar and visual depth fusion estimation system integrating semantic information according to any one of claims 1 to 6, characterized in that: The semantic segmentation module uses the pre-trained eepLabV3+ model to complete semantic segmentation.

9. A depth estimation method of laser radar and vision fusion by fusing semantic information, characterized in that: The following steps are involved: Step 1: Use the residual network to extract image features from RGB image data; Step 2: extracting point cloud features from radar point cloud data using a laser radar branch network, wherein the laser radar branch network includes a sparse pre-mapping module and a residual block connected to the sparse pre-mapping module, wherein the sparse pre-mapping module is used to extract a first point cloud feature from the radar point cloud data, and the residual block is used to extract a second point cloud feature, i.e., a final point cloud feature, from the first point cloud feature; Step 3: Use the semantic segmentation module to perform semantic segmentation on the RGB image data to obtain semantic segmentation information; Step 4: Perform cross-modal fusion of image features and point cloud features to obtain fusion features; Step 5: Based on the fused features, a sparse depth map is generated using the first decoder; Step 6: Use the second encoder to concatenate the semantic segmentation information and the sparse depth map, and then input them into the second decoder to obtain a dense depth map through decoding; Step 7: Generate the final depth prediction map based on the dense depth map; In the above steps, there is no order of precedence for steps 1 to 3.

10. The depth estimation method of laser radar and vision fusion by fusing semantic information according to claim 9, characterized in that: The step 7 also includes visualizing the final depth prediction map.

Images