A ground-free multimodal fusion 3D target detection method

Abstract

The present invention relates to a ground-free multimodal fusion 3D target detection method. The method senses obstacles in data transmitted back by sensors through a constructed target detection model. The model training process includes: based on multi-view laser radar point clouds and images, processing to obtain an image surround view splicing map and a 360° surround view splicing point cloud map, performing pixel feature extraction on the image surround view splicing map to obtain surround view image features; processing the surround view image features to obtain pseudo radar features; stacking and fusing the point cloud features and the pseudo radar features to obtain fused features; performing channel attention mechanism feature extraction on the fused features, assigning fusion weights to the point cloud and image channels, and performing classification regression as the final features; defining a quality loss function to train the target detection model. The present invention improves the accuracy and robustness of target detection. At the same time, the point cloud data is ground-free processed, which greatly reduces the network computation load, improves the detection speed, and meets the real-time requirements of unmanned driving.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving, and in particular to a ground-based multimodal fusion 3D target detection method. Background Art

[0002] 3D object detection technology uses 3D sensors (such as LiDAR) to acquire 3D information about the environment and then utilizes machine learning, deep learning, and other techniques to detect and identify objects in the scene. In the field of autonomous driving, 3D object detection technology can help autonomous vehicles perceive the surrounding traffic environment, including vehicles, pedestrians, and traffic signs on the road. This allows for autonomous navigation and safe driving, improving vehicle safety and accuracy and reducing the occurrence of traffic accidents.

[0003] There are two existing 3D object detection technologies: one is the 3D object detection technology based on pure laser point cloud, and the other is the cross-modal fusion technology of point cloud and image used in MV3D (Multi-View 3D Object Detection).

[0004] Pure laser point cloud 3D target detection technology uses multiple sensors (such as multiple lidars and cameras) to collect point cloud data in the environment, performs target detection on the point cloud data collected by each sensor separately, obtains target detection results from multiple perspectives, and finally integrates these results to obtain more comprehensive environmental perception data.

[0005] Since each sensor needs to perform target detection, it consumes a significant amount of time and computing resources. Furthermore, the point cloud data between different sensors contains a certain amount of redundancy, which results in a large amount of computation required by the neural network. Therefore, the detection speed of pure laser point cloud 3D target detection technology cannot meet the real-time requirements of autonomous driving. Furthermore, each sensor performs target detection independently, making missed detections and false detections prone to occur. The point cloud data between different sensors contains a certain amount of noise and errors, which may affect the accuracy of target detection.

[0006] The cross-modal fusion technology of point cloud and image used in MV3D achieves cross-modal object detection by stacking and fusing the features of point cloud and image. Specifically, the technical solution includes the following steps:

[0007] Feature extraction of point clouds and images: For point cloud data, features can be extracted using sliding window-based methods or voxel-based methods. For image data, convolutional neural networks (CNNs) can be used for feature extraction.

[0008] Feature stacking and fusion: The features of the point cloud and image are stacked and fused to form a unified feature representation. Specifically, bilinear interpolation or convolution operations can be used to fuse the two features.

[0009] Object detection: For the fused features, a 3D convolutional neural network (CNN) is used for object detection.

[0010] However, due to the domain gap between point clouds and images, stacking and fusing features can affect data quality and recognition results. For example, point cloud data is sparse and geometrically structured, while image data contains color and texture information. These different features can interfere with each other when stacked and fused, resulting in decreased data quality and suboptimal recognition results. Furthermore, this technical solution still requires processing large amounts of point cloud and image data, resulting in a high computational load and detection speeds that still cannot meet the real-time requirements of autonomous driving. Summary of the Invention

[0011] To solve the above problems, the present invention provides a multi-modal fusion 3D target detection method without ground.

[0012] This method uses a constructed target detection model to detect obstacles in the data transmitted by the sensor. The training process of the target detection model includes the following steps:

[0013] Step 1: Acquire multiple multi-view lidar point clouds based on multiple lidar sensors, and acquire multiple images based on multiple camera sensors;

[0014] Step 2: stitch several images together to obtain a surround stitching image; stitch the multi-view LiDAR point clouds together to obtain a 360° LiDAR point cloud, and describe the 360° LiDAR point cloud using a unified coordinate system;

[0015] Step 3: Identify and remove ground points from the 360° LiDAR point cloud to obtain a 360° surround stitching point cloud image.

[0016] Step 4: Extract point cloud features from the 360° surround stitching point cloud image to obtain point cloud features, and construct a pseudo radar feature extraction branch and a fusion feature extraction branch;

[0017] Step 5: Extract pixel features from the surround view mosaic image to obtain surround view image features;

[0018] Step 6: Input the surround view image features into the attention mask network to extract the region of interest in the image, perform depth recovery on the region of interest, predict the depth information of each pixel, and project it into three-dimensional space to generate a pseudo radar feature map;

[0019] Step 7: Input the pseudo radar feature map into the pseudo radar feature extraction branch to extract the pseudo radar feature;

[0020] Step 8: stack and fuse the point cloud features and pseudo radar features, and input the fused features into the fusion feature extraction branch to obtain the fused features;

[0021] Step 9: Use the fused features to extract channel attention mechanism features, assign fusion weights to point cloud and image channels, and use them as the final features for classification regression;

[0022] Step 10: Define a quality loss function to evaluate the quality of each predicted box and calculate the 3D bounding box confidence;

[0023] Step 11: Train the target detection model based on the quality loss function.

[0024] Furthermore, the unified coordinate system described in step 2 specifically refers to rotating the lidar point cloud under a 360° viewing angle to the vehicle body coordinate system through a calibration parameter matrix.

[0025] Furthermore, the step 4 extracts point cloud features from the 360° surround view stitched point cloud image to obtain point cloud features, specifically projecting the lidar point cloud under a 360° viewing angle into the front view, and obtaining a dense point cloud front view through a three-dimensional interpolation method, and then extracting point cloud features to obtain point cloud features.

[0026] Furthermore, the three-dimensional interpolation method specifically refers to one of a trilinear interpolation method, a cubic spline interpolation method and a cubic Hermite interpolation method.

[0027] Furthermore, the extracting of point cloud features to obtain point cloud features specifically refers to extracting point cloud features using a PointPillars backbone feature extraction network to obtain point cloud features.

[0028] Furthermore, the step 5 of extracting pixel features from the image surround stitching image specifically refers to extracting pixel features from the image surround stitching image using a pre-trained yolov3 algorithm.

[0029] Furthermore, the step of extracting the region of interest in the image in step 6 specifically refers to extracting the region of interest in the image by learning the heat map position of the obstacle feature.

[0030] Furthermore, step ten specifically includes:

[0031] Define the quality loss function P 3D for:

[0032]

[0033] Among them, N represents the number of bounding boxes in a batch, i represents the intersection-over-union ratio between the network prediction box and the true value box, and S i Represents constraints to measure the similarity between the true value box and the predicted box of each target, Represents the perception result of the quality-aware network on the quality of each predicted box.

[0034] One or more technical solutions provided in the embodiments of this application have at least the following technical effects or advantages:

[0035] This invention extracts cross-modal features and assigns weights through a multi-branch parallel feature extraction network, enabling a more comprehensive description of information in the surrounding environment, thereby improving the accuracy and robustness of target detection. Furthermore, this technology proposes quality-aware loss constraints, enabling refined evaluation and optimization of target detection results, further improving the accuracy and robustness of target detection. Furthermore, de-grounding the point cloud data significantly reduces network computational overhead, improves detection speed, and meets the real-time requirements of autonomous driving. DETAILED DESCRIPTION

[0036] The present invention is described in detail below in conjunction with specific embodiments. Before describing in detail the technical solutions of each embodiment of the present invention, the nouns and terms involved are explained. In this specification, components with the same name or the same number represent similar or identical structures and are for illustrative purposes only.

[0037] The present invention provides a multi-modal fusion 3D target detection method without ground, and the specific steps are as follows:

[0038] Step 1. Multi-view LiDAR point cloud acquisition: Multiple multi-view LiDAR point clouds are acquired based on multiple LiDAR sensors. These point clouds are described in their respective coordinate systems. Multiple images are acquired based on multiple camera sensors.

[0039] Step 2. Image stitching, point cloud stitching and coordinate system transformation: stitch several images together to obtain an image stitching map; stitch the multi-view lidar point cloud together to obtain a 360° lidar point cloud, and rotate the 360° lidar point cloud to the vehicle body coordinate system through the calibration parameter matrix. Since the multi-view lidar point clouds are described in their respective coordinate systems in step 1, it is necessary to stitch multiple multi-view lidar point clouds together to obtain a 360° point cloud. However, due to the different installation positions and orientations of each lidar sensor, the stitched point cloud is still not described in the same coordinate system, and a coordinate system transformation is required, that is, the stitched point cloud is rotated to the vehicle body coordinate system for description.

[0040] Step 3. Remove ground points from the LiDAR point cloud: Identify and remove ground points from the 360° LiDAR point cloud, while simultaneously obtaining a 360° stitched point cloud at the same timestamp. During the ground point identification step, morphological filtering from the PCL library is used to determine whether a point is a ground point, using a set filter window size and elevation difference threshold. Removing ground points yields a 360° stitched point cloud, which improves the quality and reliability of the point cloud data.

[0041] Step 4. Point Cloud Feature Extraction: Project the 360° LiDAR point cloud onto the front view and use trilinear interpolation to convert the sparse LiDAR point cloud into a dense front view. The PointPillars backbone feature extraction network is then used to extract point cloud features. Two parallel branches are derived from the PointPillars backbone feature extraction network: a pseudo-radar feature extraction branch and a fusion feature extraction branch.

[0042] Step 5. Pixel feature extraction: Use the pre-trained YOLOv3 algorithm to extract pixel features from the surround view mosaic image to obtain surround view image features, further enhancing perception and recognition capabilities.

[0043] Step 6. Obstacle heatmap positioning and depth estimation: The surround view image features are input into the attention mask network. By learning the heatmap positions of obstacle features, the region of interest in the image is extracted. Based on the camera imaging principle and the camera's internal and external parameters, the depth of the region of interest is restored. The depth information of each pixel is predicted and projected into three-dimensional space to generate a dense pseudo-radar feature map.

[0044] Step 7. Pseudo radar feature extraction: Input the pseudo radar feature map into the pseudo radar feature extraction branch to extract the pseudo radar features.

[0045] Step 8. Vision-point cloud cross-modal feature fusion: The point cloud features and pseudo-radar features are stacked and fused, and the fused features are input into the fusion feature extraction branch to extract cross-modal fusion features, thereby obtaining the features obtained by the cross-modal fusion of vision and point cloud features: fusion features.

[0046] Step 9. Classification and regression of multi-channel feature fusion: The fused features are subjected to channel attention mechanism feature extraction, fusion weights are assigned to the point cloud and image channels, and the final features are used for classification and regression.

[0047] Step 10. Design of quality-aware loss function: A quality loss function is proposed to evaluate the quality of each prediction box and calculate the confidence of the 3D bounding box. Specifically, each prediction box is scored by perceiving the quality of the 3D bounding box, and finally the confidence of the 3D bounding box is calculated by combining the confidence score of the 2D detection box.3D for:

[0048]

[0049] Among them, the parameter N represents the number of bounding boxes in a batch, i represents the Iou (intersection-over-union ratio) between the network prediction box and the true value box, and S i Represents constraints to measure the similarity between the true value box and the predicted box of each target, Indicates the perception of the quality of each predicted box.

[0050] S i Represents the similarity between the true value box and the predicted box, by constructing the cross entropy loss function P 3D , so that the network prediction value It can accurately reflect the reliability of the 3D bounding box. When the IoU (intersection over union) of two bounding boxes is the same, the quality constraint function S i At the same time, similarity constraints are designed in the algorithm, which measures the ratio of the length, width and height of the predicted box to the length, width and height of the true value box. The higher the ratio, the higher the quality of the predicted box.

[0051] Step 11. Model acceleration and deployment: Convert the trained model weights into the ONNX format. Convert the ONNX model parameters and the prototxt files of each network node into a Caffe model. Use Huawei's ATC tool to convert the Caffe model into an OM model for network acceleration. Finally, deploy the model on the mini-end of Huawei's MDC to integrate and perceive obstacles in real time from sensors in different modalities in the autonomous driving system.

[0052] The above-described embodiments are merely descriptions of preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should fall within the scope of protection determined by the claims of the present invention.

Claims

1. A multimodal fusion 3D target detection method without ground, characterized in that: The constructed target detection model is used to detect obstacles in the data transmitted by the sensor. The training process of the target detection model includes the following steps: Step 1: Acquire multiple multi-view lidar point clouds based on multiple lidar sensors, and acquire multiple images based on multiple camera sensors; Step 2: stitch several images together to obtain a surround stitching image; stitch the multi-view LiDAR point clouds together to obtain a 360° LiDAR point cloud, and describe the 360° LiDAR point cloud using a unified coordinate system; Step 3: Identify and remove ground points from the 360° LiDAR point cloud to obtain a 360° surround stitching point cloud image. Step 4: Extract point cloud features from the 360° surround stitching point cloud image to obtain point cloud features, and construct a pseudo radar feature extraction branch and a fusion feature extraction branch; Step 5: Extract pixel features from the surround view mosaic image to obtain surround view image features; Step 6: Input the surround view image features into the attention mask network to extract the region of interest in the image, perform depth recovery on the region of interest, predict the depth information of each pixel, and project it into three-dimensional space to generate a pseudo radar feature map; Step 7: Input the pseudo radar feature map into the pseudo radar feature extraction branch to extract the pseudo radar feature; Step 8: stack and fuse the point cloud features and pseudo radar features, and input the fused features into the fusion feature extraction branch to obtain the fused features; Step 9: Use the fused features to extract channel attention mechanism features, assign fusion weights to point cloud and image channels, and use them as the final features for classification regression; Step 10: Define a quality loss function to evaluate the quality of each predicted box and calculate the 3D bounding box confidence; Step 11: Train the target detection model based on the quality loss function.

2. The multimodal fusion 3D target detection method according to claim 1 is characterized in that: The unified coordinate system described in step 2 specifically refers to rotating the lidar point cloud under a 360° viewing angle to the vehicle body coordinate system through the calibration parameter matrix.

3. The multimodal fusion 3D target detection method according to claim 1 is characterized in that: The point cloud features are extracted from the 360° surround stitching point cloud image in step 4 to obtain the point cloud features. Specifically, the lidar point cloud under the 360° perspective is projected into the front view, and a dense point cloud front view is obtained by a three-dimensional interpolation method, and then the point cloud features are extracted to obtain the point cloud features.

4. The multimodal fusion 3D target detection method according to claim 3 is characterized in that: The three-dimensional interpolation method specifically refers to one of a trilinear interpolation method, a cubic spline interpolation method and a cubic Hermite interpolation method.

5. The multimodal fusion 3D target detection method according to claim 3 is characterized in that: The extracting of point cloud features to obtain point cloud features specifically refers to extracting point cloud features using a PointPillars backbone feature extraction network to obtain point cloud features.

6. The multimodal fusion 3D target detection method according to claim 1 is characterized in that: The pixel feature extraction of the image surround stitching map described in step 5 specifically refers to the pixel feature extraction of the image surround stitching map using the pre-trained yolov3 algorithm.

7. The multimodal fusion 3D target detection method according to claim 1 is characterized in that: The step 6 of extracting the region of interest in the image specifically refers to extracting the region of interest in the image by learning the heat map position of the obstacle feature.

8. The multimodal fusion 3D target detection method according to claim 1 is characterized in that: Step 10 specifically includes: Define the quality loss function P 3D for: Among them, N represents the number of bounding boxes in a batch, i represents the intersection-over-union ratio between the network prediction box and the true value box, and S i Represents constraints to measure the similarity between the true value box and the predicted box of each target, Represents the perception result of the quality-aware network on the quality of each predicted box.

Images