Color image feature and laser radar point cloud feature fusion method based on attention mechanism
Through the fusion of color images and lidar features based on attention mechanism, the sparse depth data generated by depth sensors is solved, more accurate environmental perception and more efficient depth completion are achieved, and the performance of robots and autonomous driving tasks is improved.
Patent Information
- Application Number
- CN202310840045.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-07-10
- Publication Date
- 2025-01-10
AI Technical Summary
The sparse depth data generated by existing depth sensors limits the accurate perception and understanding of the environment and affects the execution effect of robot tasks and autonomous driving, especially in tasks such as obstacle avoidance, target detection and three-dimensional scene reconstruction.
Using color image features and lidar point cloud feature fusion method based on attention mechanism, features are extracted through convolutional neural network, global pooling operation and multi-head attention mechanism are used to calculate the fusion weight, and feature fusion is realized, and used for deep completion. Generate a complete and dense depth map.
By fusing color images and lidar information, the accuracy of scene understanding and object detection is improved, the performance and effect of robot tasks and autonomous driving is improved, model errors are reduced, and inference speed is accelerated.
Smart Images
Figure CN119295867A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the fields of deep learning and color image guided depth completion, and in particular to a method for fusing laser radar point cloud features with color image features. Background Art
[0002] Accurate depth perception of the surrounding environment is critical in areas such as intelligent robotics tasks and autonomous driving. These applications require accurate perception of the environment and use depth information for task reasoning and operation execution. Depth perception plays a key role in solving tasks such as obstacle avoidance, object detection or recognition, and 3D scene reconstruction.
[0003] However, existing depth sensors have some limitations, resulting in incomplete and sparse depth data. Taking LiDAR as an example, due to the limitations of its scanning lines and scanning frequency, LiDAR can only provide a limited number of depth measurement points, resulting in sparse depth map data. This sparse depth map data limits the ability to accurately perceive and understand the environment, which may affect the performance of subsequent tasks.
[0004] The goal of the depth completion task is to estimate the depth of missing areas in a sparse depth map and generate a complete and dense depth map. Through depth completion, the gaps in sensor data can be filled, providing a more accurate depth perception of the environment, thereby providing more reliable input for subsequent tasks. In depth completion, the introduction of color image information has significant benefits. Color images provide rich color and texture information, which can be combined with sparse depth maps to further enhance perception capabilities. By fusing the semantic information of color images and the depth information of LiDAR, more accurate scene understanding, object detection, and recognition can be achieved, further improving the performance and effects of applications such as robotics tasks and autonomous driving. Summary of the invention
[0005] In order to efficiently fuse the semantic information of color images and the depth information of laser radar point clouds, the present invention provides a method for fusing color image features and laser radar point cloud features based on an attention mechanism. Through the attention mechanism, the fusion weight between the two features is calculated for feature fusion and used for subsequent depth completion. The steps include: Step S10: Use convolutional neural network to extract sparse depth image features and color image features, respectively. and Instead, their width ,high and the number of feature channels Stay consistent; Step S20: and Through the global pooling operation (MaxPooling), the width and height are both 1, and the number of feature channels is Features and Expand it into a vector form, and multiply the query weight matrix, keyword weight matrix and feature vector to get the corresponding query and keyword; Step S30: For the , , , Multiply the long matrix and Get the query and keywords in the multi-head attention mechanism; Step S40: Use the query and keyword obtained in the above step to calculate the weights in each attention head and compare them with and Weighted summation to obtain fusion features , concatenate the fusion features obtained from different attention heads and align the dimensions to , and obtain the final fusion feature . BRIEF DESCRIPTION OF THE DRAWINGS
[0006] Figure 1 It is a flow chart of the steps of the present invention.
[0007] Figure 2 It is a structural schematic diagram of the feature fusion module of the present invention.
[0008] Figure 3 This is a structural diagram of the convolutional neural network used in the present invention. DETAILED DESCRIPTION
[0009] The following describes the implementation of the present invention through specific examples and in conjunction with the accompanying drawings. Those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific examples, and the details in this specification can also be modified and changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention.
[0010] Step 1: Using the known partitioning of the KITTI Depth Completion dataset, perform steps 2 to 3 on the sparse depth map, color image, and true value provided by the training set; Step 2: The main framework structure of the neural network structure of the present invention is as follows Figure 3As shown in the figure, the neural network structure includes a feature preprocessing module, a feature encoding module, a feature decoding module and a depth prediction module. The sparse depth map and the corresponding color image are input into the feature preprocessing module of the neural network structure to extract features initially; the feature encoding module downsamples the features and performs feature fusion. Each time the feature image is downsampled, the height and width are 1 / 2 of the original, and the number of feature channels becomes twice the original; the feature decoding module needs to restore the resolution of the feature encoding module features, and the feature image output by the feature encoding module is matched through channels through 1×1 convolution. Before sending the feature image to the next layer, the deconvolution result needs to be stacked with the processed sparse depth map through channels, and a total of 4 upsampling is performed; finally, the depth prediction module uses 1×1 convolution to The characteristic image is restored to The feature encoding module is composed of a fusion module designed by the present invention, which is stacked in accordance with ResNet18 and ResNet34 and named FRNet18 and FRNet34. Figure 2 As shown, the number of attention heads is 8; Table 1 Convolution structure of feature preprocessing module: Convolution kernel size Step Length Completion Number of input channels Number of output channels Sparse Depth Image 3X3 2 1 1 16 Color images 3X3 2 1 3 48 For the feature preprocessing module in step 1, the convolution structure is detailed in Table 1, and its specific implementation is as follows: 1. Expand the channels of the two images in a ratio of 3:1 and then superimpose the feature maps; 2. Add BN layer and ReLU layer after the convolution layer; The feature fusion module in step 2 is the same as the ResNet18 / 34 network structure, except that the residual block is replaced by the fusion module of the present invention. The resolutions of the feature maps output in the four stages are , , , , the specific structure is shown in Table 2, and the specific implementation method is as follows: Table 2 Feature fusion module structure table:
[0011] 1. The input features are color image features and sparse depth image features, which are extracted by convolution with a 3×3 convolution kernel and a step size of 2, respectively. and ; 2. and Through the global pooling operation (MaxPooling), the width and height are both 1, and the number of feature channels is Features and Expand it into a vector form, and multiply the query weight matrix, keyword weight matrix and feature vector to get the corresponding query and keyword;
[0012] in, , , , They are respectively the query obtained by color image features, the query obtained by sparse deep image features, the keyword obtained by color image features, and the keyword obtained by sparse deep image features; 3. For the above steps , , , Multiply the long matrix and Get the query and keywords in the multi-head attention mechanism;
[0013] in, , , , Representative The query and keywords in each attention head; 4. Use the query and keyword obtained in the above steps to calculate the weights in each attention head and compare them with and Weighted summation to obtain fusion features , concatenate the fusion features obtained from different attention heads and align the dimensions to , and obtain the final fusion feature ;
[0014]
[0015] Step 3: For the regressed depth residual map outputted in step 2, supervised learning is performed using the true depth map contained in the training set. A total loss function is set to calculate the total loss for each valid pixel in the true depth map, where a valid pixel is a pixel with a depth value. The specific training process is as follows: using the Adam optimizer with a batch size of 4. The initial learning rate is set to 0.001. After 10 rounds of training, the entire network has converged.
[0016] Step 4: Using the sparse depth truth values in the validation set and the test set, the neural network structure of all parameters that have been trained in step 3 is quantitatively tested.
[0017] A total of four models were trained during testing, and the model structure is Figure 3 The structure shown in the figure replaces the encoder module with the network structure of the encoder stage in ResNet18, ResNet34, FRNet18, and FRNet34 respectively. From the results shown in Table 3 below, it can be seen that the fusion module designed by the present invention reduces the error of the model and also speeds up the reasoning speed of the model. Since this paper adopts a modular design, the network structure of the encoder stage of FRNet18 and FRNet34 can be used to replace the downsampling modules of some other single-branch models.
[0018] Table 3 Model evaluation results on the test set: Feature Encoder RMSE(mm) MAE(mm) Inference time (ms) ResNet18 1015.826 319.965 55.438 ResNet34 952.414 322.218 91.258 FRNet18 903.822 280.910 42.721 FRNet34 889.539 267.488 65.405
Claims
1. A method for fusing color image features and lidar point cloud features based on an attention mechanism, characterized in that: The attention mechanism is used to calculate the similarity between features, and the fusion features are obtained according to the similarity. Finally, the fusion features are used to complete the depth of the sparse depth image to obtain a dense depth image, and the following steps are included: S10: Use convolutional neural networks to extract sparse depth image features and color image features with consistent width, height, and number of channels; S20: compressing and expanding the two features obtained in step S10 into vector form through a global pooling operation, and calculating the query and keyword of the two features using a query weight matrix (Q) and a keyword weight matrix (K); S30: Calculate the similarity between features using the attention mechanism, use the query and keyword obtained in step S20 to calculate the similarity, and calculate the matching feature fusion weight according to the similarity; S40: Perform weighted summation on the two features obtained in step S10 and the weight obtained in step S30 to obtain a fused feature.
2. According to the method for fusing color image features and laser radar point cloud features based on attention mechanism in claim 1, it is characterized in that: Following the ResNet feature extractor part, the fusion modules described in claim 1 are stacked to obtain a feature extraction backbone network.
3. According to the method for fusing color image features and laser radar point cloud features based on attention mechanism in claim 1, it is characterized in that: In the steps S20 and S30, a multi-head attention mechanism is adopted, and multiple groups of queries and keywords are used to calculate feature fusion weights.