Multi-sensor fusion perception method for autonomous driving scenarios

Abstract

The present invention relates to the field of computer vision technology, and specifically to a multi-sensor fusion perception method for autonomous driving scenarios. The method uses multi-perspective images and radar point cloud data as input, extracts features through image and radar processing networks respectively, performs secondary feature fusion at the feature level, and ultimately realizes 3D target detection. The present invention obtains radar semantic features and radar target probability map information through radar point cloud data, guides the generation of image bev features by fusing radar target probability map information, and improves the image feature's ability to express targets. A secondary feature fusion strategy is adopted, by fusing radar semantic features and image bev features, and using a bev encoder to further extract multi-sensor data features, which can more fully fuse multi-sensor features and improve the network's ability to model targets.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a multi-sensor fusion perception method for autonomous driving scenarios. Background Art

[0002] Currently, 3D object detection is a hot research topic, attracting widespread attention from both academia and industry. Depending on the sensor used, 3D object detection algorithms can be categorized as camera-based or fusion-based. Traditional camera-based 3D object detection algorithms have benefited from the development of 2D detectors. These algorithms expand upon these 2D detectors and add prediction tasks to achieve 3D object detection. Methods such as FCOS3D and DETR3D achieve 3D object prediction by adding predictions for the 3D object's orientation angle, height, and center point spatial position.

[0003] However, predicting 3D objects from 2D images is inherently an ill-posed problem, and the lack of depth information leads to low prediction accuracy in these methods. Multi-sensor fusion algorithms, primarily LiDAR-Camera-based, use laser pulses to scan the environment rather than radio waves or ultrasound. The smaller wavelength means LiDAR systems can detect smaller objects with high resolution and accuracy. By fusing LiDAR's high-precision spatial information with the rich semantic information of images, they improve 3D object detection performance.

[0004] However, LiDAR is susceptible to poor weather and lighting conditions, and its high price makes it unsuitable for industrial mass production. Radar offers all-weather operation, a long detection range, and a relatively low price. Consequently, radar-camera fusion detection methods have also been developed, such as Centerfusion and Craft. These methods are decision-level fusion methods that first predict target information using images and radar networks separately. Post-processing methods then de-associate and match targets, producing a final prediction result. While these methods have achieved some performance, they are computationally intensive, have complex post-processing rules, and fail to fully leverage the characteristics of multi-sensor fusion. Summary of the Invention

[0005] In response to the problems existing in the background technology, the present invention proposes a multi-sensor fusion perception method for autonomous driving scenarios, which extracts multi-view image features, radar features and target probability map information respectively. It adopts a secondary fusion method, first guiding the generation of image bev features by fusing image features and target probability map information, and then further fusing radar and image bev features. This is a feature-level fusion strategy that can fully integrate the features of multiple sensors and improve detection accuracy.

[0006] To achieve the above object, the present invention provides the following technical solutions:

[0007] A multi-sensor fusion perception method for autonomous driving scenarios uses multi-view images and radar point cloud data as input, extracts features through image and radar processing networks respectively, performs secondary feature fusion at the feature level, and ultimately achieves 3D target detection.

[0008] Furthermore, the processing of the image data specifically includes the following steps:

[0009] (1.1) Input multi-visual image data;

[0010] (1.2) Preprocess the image;

[0011] (1.3) Extract 2D image features;

[0012] (1.4) Perform feature fusion on the extracted features;

[0013] (1.5) Setting a discrete depth information prediction branch to predict the discrete depth information of each feature map pixel;

[0014] (1.6) 2D to 3D feature conversion: Based on the predicted depth information and camera internal and external parameters, the coordinates of the 2D feature points corresponding to the 3D space are obtained according to the coordinate conversion formula, that is, the feature representation of the 2D feature in the 3D space is obtained;

[0015] (1.7) BEV-pooling module: Based on the ego-vehicle coordinate system, a 128*128 bev grid is pre-divided, with each grid size of [0.5m, 0.5m]. Based on the divided grids, points in the 3D space are assigned to the corresponding grids, and their feature representation in the bev space is obtained through sum-pooling;

[0016] (1.8) First fusion: The object probability map in the BEV space obtained from radar data is fused with the image BEV features, and the target probability predicted by radar is used to guide the further extraction of image BEV features;

[0017] (1.9) Second fusion: The extracted image bev features and radar semantic features are concatenated along the channel dimension, and the concatenated features are further fused through the bev encoder;

[0018] (1.10) Prediction head: Predicts the length, width, height, center point position, orientation angle, speed, and category information of the target through discrete prediction branches.

[0019] Preferably, the image preprocessing in step (1.2) is performed by using a random flipping, scaling, and rotation preprocessing method.

[0020] Preferably, the feature fusion in step (1.4) is performed through an FPN (feature pyramid network) network.

[0021] Furthermore, the conversion formula of step (1.6) is shown as (1):

[0022]

[0023] where d i is the predicted depth value, K is the camera internal parameter, R is the rotation matrix from the camera coordinate system to the world coordinate system, t is the translation matrix, z i When the depth value of the 2D pixel is d i The corresponding 3D coordinate height value.

[0024] Furthermore, the processing of the radar point cloud data specifically includes the following steps:

[0025] (2.1) Voxelization: First, the irregular point cloud data is voxelized to obtain radar point cloud features;

[0026] (2.2) Radar feature extraction network, obtaining the BEV feature representation of radar data;

[0027] (2.3) Radar bev feature extraction, further extract radar features;

[0028] (2.4) Extraction of radar target overview and semantic features: Through the decoupled radar prediction branch, the radar target probability map and semantic feature representation are obtained, and then they are secondary fused with the image features.

[0029] Preferably, the specific steps of step (2.2) are to perform max pooling on the point cloud data along the feature dimension through a fully connected layer network to obtain voxel features, and assign the features to the corresponding grids according to the divided bev grids to obtain the bev feature representation of the radar data.

[0030] Preferably, the step (2.3) further extracts radar features through ResNet-18 and FPN structures.

[0031] Compared with the prior art, the present invention has the following beneficial effects:

[0032] This paper designs a new radar-camera fusion network, which can improve the target detection accuracy in complex scenes, fully utilize the complementary characteristics of multiple sensors, and enhance the robustness of the algorithm.

[0033] This method uses radar point cloud data to obtain radar semantic features and radar target probability map information. By integrating this information with the radar target probability map, it guides the generation of image BEV features, improving the image features' ability to express the target. A secondary feature fusion strategy is employed to fuse radar semantic features with image BEV features. A BEV encoder is then used to further extract multi-sensor data features, enabling more comprehensive integration of multi-sensor features and improving the network's ability to model targets. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The drawings described herein are used to provide a further understanding of the embodiments of the present invention, constitute a part of this application, and do not constitute a limitation of the embodiments of the present invention. In the drawings:

[0035] Figure 1 Flow chart of the detection method of the present invention.

[0036] Figure 2 This is a diagram of the radar point cloud data processing network structure of the present invention.

[0037] Figure 3 This is a visualization diagram of some detection results of the embodiments of the present invention. DETAILED DESCRIPTION

[0038] In order to further illustrate the technical means and effects adopted by the present invention to achieve the predetermined purpose of the invention, the specific implementation methods, structures, features and effects of the present invention are described in detail below in conjunction with the accompanying drawings and preferred embodiments.

[0039] like Figure 1 and Figure 2 As shown in the figure, the present invention presents a multi-sensor fusion perception method for autonomous driving scenarios. It uses multi-view images and radar point cloud data as input, extracts features through image and radar processing networks, and fuses them at the feature level to ultimately achieve 3D object detection. The algorithm detection process is divided into two main lines: the first is image data processing, and the second is radar point cloud data processing. These two main lines are described in detail below.

[0040] 1. Image data processing

[0041] Step 0: Input multi-view image data.

[0042] Step 1: Image preprocessing. This method uses random flipping, scaling, and rotation for preprocessing.

[0043] Step 2: 2D image backbone network. In this method, ResNet-50 is used as the backbone network to extract the features of 2D images.

[0044] Step 3: Feature fusion network. In this method, the extracted features are fused through the feature pyramid network (FPN).

[0045] Step 4: Discrete depth information prediction branch. This branch predicts the discrete depth information of each feature map pixel.

[0046] Step 5: 2D-to-3D feature conversion module. Based on the predicted depth information and the internal and external parameters of the camera, the coordinates of the 2D feature points corresponding to the 3D space are obtained according to the coordinate conversion formula, that is, the feature representation of the 2D feature in the 3D space is obtained. The conversion formula is shown in (1). i is the predicted depth value. In this method, the predicted depth value range is [1m, 60m] with an interval of 1m. K is the camera internal parameter, R is the rotation matrix from the camera coordinate system to the world coordinate system, t is the translation matrix, and z i When the depth value of the 2D pixel is d i The corresponding 3D coordinate height value.

[0047]

[0048] Step 6: BEV-pooling module. Pre-divide the vehicle coordinate system into a 128x128 BEV grid, with each grid size of [0.5m, 0.5m]. Based on the divided grids, assign points in 3D space to corresponding grids and use sum-pooling to obtain their feature representation in BEV space.

[0049] Step 7: First fusion. The probability map of the object in the BEV space is extracted based on the radar data and fused with the image BEV features. The target probability predicted by the radar is used to guide the further extraction of the image BEV features.

[0050] Step 8: Second fusion. The extracted image bev features and radar semantic features are concatenated along the channel dimension, and the concatenated features are further fused through the bev encoder.

[0051] Step 9: Prediction head. Through discrete prediction branches, the length, width, height, center point position, orientation angle, speed, and category information of the target are predicted.

[0052] 2. Radar data processing

[0053] Step 1: Voxelization: First, the irregular point cloud data is voxelized to obtain radar point cloud features.

[0054] Step 2: Radar feature extraction network. A fully connected layer network performs maxpooling on the point cloud data along the feature dimension to obtain voxel features. Based on the divided BEV grid, the features are assigned to the corresponding grids to obtain the BEV feature representation of the radar data.

[0055] Step 3: Radar bev feature extraction: Radar features are further extracted using the ResNet-18 and FPN structures.

[0056] Step 4: Extraction of radar object probability map and semantic features. Through the decoupled radar prediction branch, the radar target probability map and semantic feature representation are obtained, and then they are fused with the image features for a second time.

[0057] Example: 3D target detection experiment using the multi-sensor fusion perception method (hereinafter referred to as RCFusionNet) for autonomous driving scenarios of the present invention

[0058] 1) Comparison of test results

[0059] This experiment uses the nuScenes dataset, which contains 1,000 scenes, each lasting 20 seconds. The dataset includes 1.4M images, 1.4M frames of radar data, and 1.4M bounding boxes. These bounding boxes are 3D detection boxes for 23 categories, annotated at 2Hz. The annotations include the category, the translation matrix (in meters) in the global coordinate system, the rotation matrix (in meters) represented by quaternions, and the 3D dimensions (in meters).

[0060] Table 1 shows the performance comparison of this method and other detection methods. Our method RCFusionNet achieved 37.3% on mAP and 46.6% on NDS. In particular, compared with BEVDet using the Swin-T backbone network, our method improved by 6.1% on mAP and 7.4% on NDS, even though we used a relatively weak 2D backbone network. In addition, compared with the BEVDepth method, our method improved by 2.2% on mAP. Furthermore, when we used ResNet101 as the backbone network, by comparing the performance changes brought about by different backbone networks, the model achieved 47.5% on NDS and 38.1% on mAP. This result shows that using a stronger 2D image feature backbone network can effectively improve the performance of the model.

[0061] Table 1-1 Comparison of detection performance of this method and other methods

[0062]

[0063] like Figure 3 Shown are some detection results from our method. In the visual comparison of results from the BEV perspective, we present the detection results of the baseline, RCFusionNet-C, and RCFusionNet, respectively. RCFusionNet-C represents the version of our method that uses only the image sensor. The yellow solid box represents the prediction result, and the blue solid box represents the ground truth annotation box. Compared with the baseline and RCFusionNet-C without radar data fusion, our method performs better in detecting long-range targets. A zoomed-in image of the local detection result (indicated by the orange box) shows that four real-world targets are present. The baseline model predicted six targets, including two false detections; RCFusionNet-C predicted five targets, including one false detection. This comparison clearly demonstrates that improvements to the image network have significantly improved performance. RCFusionNet successfully detected four targets, and the predicted target sizes and orientations were closer to the actual values, further verifying that radar data fusion significantly improves detection performance.

[0064] The above description is merely a preferred embodiment of the present invention and does not constitute any form of limitation to the present invention. Although the present invention has been disclosed as above in terms of a preferred embodiment, it is not intended to limit the present invention. Any person skilled in the art can, without departing from the scope of the technical solution of the present invention, make some changes or modifications to equivalent embodiments using the technical contents disclosed above. However, any brief modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention are still within the scope of the technical solution of the present invention.

Claims

1. A multi-sensor fusion perception method for autonomous driving scenarios, characterized by: Using multi-view images and radar point cloud data as input, features are extracted through image and radar processing networks respectively, and secondary feature fusion is performed at the feature level to ultimately achieve 3D object detection. Specifically, the detection process is divided into two main lines: one is the processing of image data, and the other is the processing of radar point cloud data. The steps for processing the image data are: (1.1) Input multi-visual image data; (1.2) Preprocess the image; (1.3) Extract 2D image features; (1.4) Perform feature fusion on the extracted features; (1.5) Setting up a discrete depth information prediction branch to predict the discrete depth information of each feature map pixel; (1.6) 2D to 3D feature conversion: Based on the predicted depth information and the internal and external parameters of the camera, the coordinates of the 2D feature points corresponding to the 3D space are obtained according to the coordinate conversion formula, that is, the feature representation of the 2D feature in the 3D space is obtained; the coordinate conversion formula is shown in (1): in is the predicted depth value, K is the camera internal parameter, R is the rotation matrix from the camera coordinate system to the world coordinate system, t is the translation matrix, When the depth value of the 2D pixel is The corresponding 3D coordinate height value; (1.7) BEV-pooling module: Based on the ego-vehicle coordinate system, a 128*128 BEV grid is pre-divided, with each grid size of [0.5m, 0.5m]. Points in 3D space are assigned to corresponding grids based on the divided grids, and their feature representation in BEV space is obtained through sum-pooling. (1.8) First fusion: The target probability map in the BEV space obtained from radar data is fused with the image BEV features, and the target probability predicted by radar is used to guide the further extraction of image BEV features; (1.9) Second fusion: The extracted image BEV features and radar semantic features are concatenated along the channel dimension, and the concatenated features are further fused through the BEV encoder. (1.10) Prediction head: Predicts the length, width, height, center point position, orientation angle, speed, and category information of the target through discrete prediction branches; The processing steps of the radar point cloud data are as follows: (2.1) Voxelization: First, the irregular point cloud data is voxelized to obtain radar point cloud features; (2.2) Max pooling is performed on the point cloud data along the feature dimension through a fully connected layer network to obtain voxel features. The features are then assigned to the corresponding grids according to the divided BEV grids to obtain the BEV feature representation of the radar data. (2.3) Radar bev feature extraction, further extracting radar features; (2.4) Extraction of radar target overview and semantic features: Through the decoupled radar prediction branch, the radar target probability map and semantic feature representation are obtained, and then they are secondary fused with the image features.

2. The multi-sensor fusion perception method for autonomous driving scenarios according to claim 1, characterized in that: The image preprocessing in step (1.2) is performed by using a random flipping, scaling, and rotation preprocessing method.

3. The multi-sensor fusion perception method for autonomous driving scenarios according to claim 1, characterized in that: The feature fusion in step (1.4) is performed through the FPN network.

4. The multi-sensor fusion perception method for autonomous driving scenarios according to claim 1, characterized in that: The step (2.3) further extracts radar features through the ResNet-18 and FPN structures.

Images