Multi-level multi-attention target detection method based on radar and camera

Abstract

The invention discloses a multi-level multi-attention target detection method based on a radar and a camera, and the method comprises the steps: firstly obtaining a radar point cloud and an image of a public data set, and carrying out the preprocessing; secondly, constructing an SC-DLA34 feature extraction network based on a DLA34 network to obtain an associated point cloud, and performing multi-level fusion on the associated point cloud and the image to obtain a preliminary fusion feature; and then splicing the associated point cloud features and the preliminary fusion features at a channel level, and inputting the spliced features into a PEA pyramid attention model to obtain deep fusion features. Secondly, the deep fusion features are sent to a 3D detection head to be decoded, the depth, speed, rotation and attribute information of a detection target is estimated, and a target 3D detection frame is generated; according to the invention, cross-modal fusion of radar information and image information is realized, the expression capability of point cloud is improved, and the detection capability of the model on small targets is ensured.

Description

Technical Field

[0001] The present invention belongs to the field of multi-sensor fusion perception, and specifically relates to a multi-level and multi-attention target detection method based on radar and camera. Background Art

[0002] Multi-sensor fusion perception is one of the core technologies in autonomous driving and intelligent robots. It is the key to perceiving the external world. Its collaborative performance directly determines the safety of pedestrians, vehicles, and machines. The detection capabilities and reliability of different sensors in different environments are limited. Multi-sensor fusion can improve the accuracy of target detection and recognition.

[0003] Cameras are one of the earliest sensors used in perception systems and are also used to complete tasks such as target recognition, environmental map construction, lane detection, and target tracking. Compared with millimeter-wave radars, the advantage of cameras is that they can accurately capture contour, texture, and color distribution information, which helps to classify and identify different targets under non-extreme environmental conditions. However, considering that in some cases, images may become unreliable, such as cameras are prone to miss detection in the case of occlusion, and the ability to detect occluded objects is insufficient, which requires other sensors to make up for the shortcomings of cameras. Compared with cameras, millimeter-wave radars have longer wavelengths and have certain anti-blocking and anti-pollution capabilities, and can cope with rain, snow, fog, and dark environments. Radar can not only obtain the precise distance of multiple targets, but also measure relative speed using the Doppler shift effect. The disadvantage of millimeter-wave radars is that it is difficult to distinguish relatively stationary targets, and traditional millimeter-wave radars have no height information and are difficult to distinguish objects with height. As a perception model applied to outdoor mobile robots, it is necessary to consider the impact of actual scenes, ensure its perception accuracy, and consider cost control. Millimeter-wave radar is a very good choice.

[0004] For multi-sensor fusion models, neural network-based fusion algorithms can be divided into data-level fusion, feature-level fusion, and decision-level fusion according to the different input data of the fusion network. Feature-level fusion can mine relevant feature information in the original image, increase the credibility of feature information, exclude false features, and establish new composite features. The amount of feature space data after feature-level fusion is greatly reduced compared to the original image data. Although feature-level fusion sacrifices some information, it achieves effective information compression, facilitates real-time processing, and ensures high accuracy. It can also use the current popular deep learning method to effectively improve the detection accuracy. Summary of the invention

[0005] Based on the above analysis, this paper fuses millimeter-wave radar point clouds and images for 3D target detection through feature-level fusion to solve the following problems in the current multi-sensor fusion perception method: the two sensors cannot fully utilize the effective features in point clouds and images, and the feature images of a single level are difficult to effectively characterize objects of various scales; the simple method of splicing and fusing different features cannot fully utilize the feature information of each sensor, and there is a gap in the amount of information of the two sensors, and the perception ability of the sensor with less information will be limited; multi-level fusion features used in target detection networks will lack channel and spatial position information.

[0006] In order to solve the above problems, the present invention provides a multi-level multi-attention target detection method based on radar and camera. The overall target detection scheme includes a multi-level fusion network and a pyramid attention fusion network. The steps are as follows:

[0007] Step 1: Get the radar point cloud and image parts from the public dataset and perform preprocessing. Perform random rotation data enhancement on the image, and use pillar expansion to enhance the representation ability of the radar point cloud. Each radar point cloud is expanded into a column with dimensions of 0.2, 0.2, and 1.5 meters in the x, y, and z axes, respectively, so that the point cloud information of the detected object is associated with the dimension in the three-dimensional space. So far, we have completed the preprocessing of the above two types of data. Next, we need to extract and fuse the features of the two databases.

[0008] Step 2: Build the SC-DLA34 feature extraction network based on the DLA34 network.

[0009] Obtaining the associated point cloud DLA34 is an image feature extraction network. This tree-like network structure can iteratively fuse the feature information of the network structure. The DLA34 network will realize the target feature extraction between different levels through downsampling. The present invention improves the traditional pooling method in the above-mentioned downsampling process into soft pooling, and replaces the convolution in the process with a learnable adaptive deformable convolution, which is called SC-DLA34 feature extraction network. Soft pooling weights each local area of ​​the feature map by using the softmax function to retain more information. The core idea of ​​the deformable convolution is to introduce a learnable offset in the traditional convolution operation. This offset is used to adjust the sampling position of the convolution kernel, so that the convolution kernel can better fit the edge and shape of the target. The present invention uses SC-DLA34 as a feature extraction network for extracting image features after preprocessing, and obtains the 2D detection frame of the target through a first-level feature detection head. The first-level detection head outputs three heads, namely, heat map features, center point position, and width and height of the detection target. The frustum association strategy proposed in Centerfusion is adopted to predict the 3D frustum of the region of interest using the 2D detection box and the depth information of the preprocessed point cloud. The point cloud data outside the region of interest is ignored to obtain the associated point cloud within the region of interest. Next, the associated point cloud and image will be multi-level fused.

[0010] Step 3: Perform multi-level fusion on the associated point cloud and the preprocessed image to obtain preliminary fusion features.

[0011] The point cloud heat map generated according to the target depth of the associated point cloud, the x-axis component and the y-axis component of the target radial velocity are respectively used as the three channels of the associated point cloud features. The associated point cloud features will be input into the multi-level fusion network with the image features extracted by the SC-DLA34 feature extraction network. The present invention proposes a cross-spatial attention model that uses the radar spatial attention weight obtained by processing the associated point cloud features through the spatial attention network. The features obtained by guiding the image features with the radar attention weights are passed through the multi-scale channel attention model MSCAM, which extracts channel attention through two branches with different scales. The local attention branch pays attention to the scale problem of the channel through point-by-point convolution, while the global attention branch obtains global context information through adaptive average pooling. The features processed by radar spatial attention and MSCAM are called new image features. These new image features are obtained by the image features output by the previous DLA34 tree structure and the associated point cloud features through the cross-spatial attention model, and will be used for the input of the next DLA34 tree, and so on. Finally, the DLA34 model will output a preliminary fusion feature of a point cloud and an image. This preliminary fusion feature is an image feature guided by the radar spatial attention. Such processing can effectively improve the camera recognition ability and robustness. This feature will be used for deeper fusion with the point cloud feature.

[0012] Step 4: Concatenate the associated point cloud features with the preliminary fusion features at the channel level, and input the concatenated features into the PEA pyramid attention model to obtain deep fusion features.

[0013] The present invention proposes a PEA pyramid attention model to further fuse the associated point cloud features with the preliminary fusion features to obtain deep fusion features. PEA divides the feature channel into four parts, passes through four deep convolutional network branches respectively, obtains four different scale features, and then splices the four scale features to obtain pyramid features. The pyramid feature directly obtains the attention weight through a convolution layer after the global average pooling operation, and finally the attention weight is multiplied by the feature tensor. The PEA model can combine multi-scale features under different receptive fields, and use convolution instead of fully connected layers to reduce a large number of parameters and ensure the real-time performance of the model. The associated point cloud features and the preliminary fusion features are spliced ​​at the channel level, and the spliced ​​features are input into the PEA pyramid attention model to obtain deep fusion features.

[0014] Step 5: Send the deep fusion features to the 3D detection head for decoding, estimate the depth, speed, rotation and attribute information of the detection target, and generate the target 3D detection frame.

[0015] Beneficial effects of the present invention:

[0016] The method proposed in the present invention can effectively realize 3D target detection. Among them, the method of fusing millimeter-wave radar point cloud and camera multi-level and multi-attention proposed in the present invention allows sparse point cloud features to guide image features in a learnable way through the attention mechanism, so that radar information and image information are fused cross-modally. The expression ability of point cloud is improved, providing stronger robustness for the camera. The PEA attention pyramid feature fusion network is used instead of simple feature splicing, so that the feature fusion part can be learned to obtain a more accurate fusion feature expression. At the same time, the backbone network is improved to have a larger receptive field, ensuring the model's ability to detect small targets. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 The target detection network structure used by the 3D target detection network of the present invention;

[0018] Figure 2 This is a structural diagram of the cross-spatial attention mechanism of the 3D object detection network of the present invention;

[0019] Figure 3 It is a structural diagram of the pyramid feature fusion module of the 3D object detection network of the present invention;

[0020] Figure 4This is a graph of actual target detection results of the 3D target detection network of the present invention;

[0021] Figure 5 This is the actual 3D detection block diagram of the 3D object detection network of the present invention. DETAILED DESCRIPTION

[0022] The present invention is further described in conjunction with a specific implementation process, and the specific implementation steps are as follows:

[0023] The overall target detection process of this method is as follows Figure 1 As shown in the figure, a multi-level multi-attention target detection method based on radar and camera. The dataset used is Nuscenes. The Nuscenes dataset is a public large-scale dataset for autonomous driving developed by the Motional (formerly nuTonomy) team. It uses common sensors such as millimeter-wave radar, lidar, and color cameras to collect 1,000 driving scenes in Boston and Singapore, two cities known for their dense traffic and challenging driving environments. The scenes collected include scenes under various road conditions and lighting conditions such as cities and suburbs. Nuscenes is the first large-scale dataset that provides data from the entire sensor suite of an autonomous vehicle (6 cameras, 1 lidar, 5 radars, GPS, IMU). Compared with KITTI, Nuscenes contains 7 times more object annotations.

[0024] Radar point cloud preprocessing and association:

[0025] Each radar is considered as a point in a 3D coordinate system, and its parameters include position information on the x, y, and z axes, as well as the component velocities Vx and Vy along the x and y axes. For each scene, the three scans performed by the millimeter-wave radar in the past 0.25 seconds are used as the point cloud data for this detection. The Nuscenes dataset provides the calibration parameters required to map the radar point cloud from the radar coordinate system to the image coordinate system and the camera coordinate system. We can map the millimeter-wave radar point cloud data to any coordinate system based on these two sets of parameters. Since the millimeter-wave radar cannot provide accurate height information, we will use the pillar extension method to increase the millimeter-wave point cloud information. The radar points are expanded into columns with dimensions of 0.2, 0.2, and 1.5 meters in the x, y, and z axes, respectively, so that the point cloud information of the detected object is associated with the dimensions in the three-dimensional space. This new point cloud representation method can give a better representation of the actual object during detection.

[0026] A cone-based association strategy is adopted. In the training phase, the true value of the object's 3D bounding box is used to create a region of interest cone ROI. In the test phase, the 2D bounding box detected by CenterNet and its estimated depth information are used to predict the 3D region of interest cone ROI. The size of the cone is controlled by a parameter. The existence of the cone reduces the detection range of the millimeter-wave radar data that needs to be checked for association, and the point cloud data outside the region of interest can be ignored. When the millimeter-wave radar detection point and the bounding box generated by the image detection are successfully associated, the depth and speed information of each successfully associated radar detection should generate a three-channel heat map centered on the two-dimensional bounding box as a supplementary feature of the image. The peak of the heat map is the normalized depth of the target point and the speed in the x and y directions.

[0027] The present invention uses the improved SC-DLA34 as the backbone feature extraction network to extract image features to generate a preliminary detection frame. This paper improves the original DLA34 network structure, uses a combination of deformable convolution and soft pooling to strengthen the extraction of image feature mirror information, and uses the CBAM hybrid attention mechanism to process the output features.

[0028] After the radar detection is associated with the target, in order to make full use of the depth and velocity information of the radar point cloud to complement the image information, the x-axis and y-axis components of the object's depth and velocity are used as heat map values ​​to generate a heat map centered on the center point of the object's 2D bounding box, which is fused with the original features as an additional channel. The point cloud and image are associated on the 112*200 feature map, and the fused features for detection are output. In order to increase the representation capability of the point cloud and give full play to the advantages of the good robustness of the millimeter-wave radar, a cross-spatial attention mechanism based on radar heat map features is proposed by using the characteristics of radar spatial attention with multi-scale spatial receptive fields. Figure 2 As shown in the figure, radar spatial attention is used to cross-fuse the input features of each tree of the DLA34 network. In order to make the associated radar feature heat map the same size as the input image of the DLA34 backbone layer, we choose to use the bilinear interpolation method for upsampling and the maximum pooling method for downsampling, and fully integrate the radar features associated with the image with each layer of the backbone network, thereby increasing the representation capability of the millimeter-wave radar point cloud. The MSCAM multi-scale channel attention module is used to learn global and local context features by focusing on the channel scale problem of the cross-fusion features through point-by-point convolution.

[0029] The associated point cloud features and the preliminary fusion features are spliced ​​at the channel level, and the spliced ​​features are input into the PEA pyramid attention model to obtain deep fusion features. The PEA pyramid attention model divides the feature channel into four parts, such as Figure 3As shown in the figure, four different scale features are obtained through four deep convolutional network branches respectively, and then the four scale features are concatenated to obtain pyramid features. The pyramid features are directly passed through a convolution layer to obtain attention weights after the global average pooling operation, and finally the attention weights are multiplied by the feature tensor.

[0030] The main indicators for evaluating the detection task of the Nuscenes dataset include: mean average precision (mAP) and Nuscenes detection score (NDS). NDS is calculated based on mAP and the average metric mTP, where the mTP indicator includes five subcategories: average translation error (ATE), average scale error (ASE), average angle error (AOE), average velocity error (AVE) and average attribute error (AAE).

[0031] The proposed method was tested on the 3D benchmark of Nuscenes, and the NDS score reached 46.9%, and the mAP score reached 34.3%. Compared with the methods of image detection, lidar detection, camera and lidar fusion detection, camera and millimeter wave radar fusion detection, etc., as shown in Table 1, the proposed method has significant improvements. All the following Centerfusion data were obtained in the aforementioned environment and using the same hyperparameters. Compared with the single-camera detection methods MonoDIS and CenterNet, the mAp score of the proposed detection method increased by 3.9% and 3.7% respectively, compared with the single radar plus detection method PointPillars, it increased by 3.8%, and compared with the multi-sensor fusion methods F-PointNet and CenterFusion, it increased by 1.7% and 1.9% respectively.

[0032] The NDS score increases by 9.4%, 2.5%, and 8.3% compared to MonoDIS, CenterNet, and InfoFocus, respectively, and increases by 1.6%, 2.9%, and 2.5% compared to PiontPillars, F-PoineNet, and CenterFusion, respectively.

[0033] Table 1

[0034]

[0035] The present invention compares the detection accuracy of the model in this paper in various target categories, and the results are shown in Table 2. Compared with the base model Centerfusion, this paper has improved the detection accuracy of smaller targets such as Motor, Pedestrain and Bicycle by 1.3%, 0.6% and 4.4% respectively. For larger target categories such as Car, Truck, Bus and Trailer, they have increased by 1.0%, 1.4%, 2.6% and 6.4% respectively. This is because the backbone of the model in this paper uses soft pooling on the network for downsampling and combines the attention mechanism to effectively expand the receptive field. At the same time, multi-level radar feature fusion enhances the robustness of target detection. And by introducing a feature fusion method based on an improved attention mechanism instead of simply splicing feature tensors at the channel layer, the accuracy of target detection is further improved.

[0036] Table 2

[0037]

[0038] The present invention conducts ablation experiments on multi-level radar fusion, feature fusion mode and backbone network improvement. The results are shown in Table 3. Figure 4 and Figure 5 As shown. This paper uses the network that does not contain the three modules, that is, the original Centerfusion model, as a control. First, soft pooling is added to the downsampling module of the backbone network, and the CBAM attention mechanism is used to process image features. By optimizing the backbone network in this way, the accuracy is improved by 0.4% compared with the baseline, proving that the improved backbone network improves the accuracy of the model. Subsequent experiments are conducted on the improved backbone network. Then, the effects of the addition of the PEA module and the Cross Spatial Attention module on the accuracy are compared, respectively. The accuracy is improved by 0.8% and 1.1% compared with the baseline, and by 0.4% and 0.7% compared with the model that only improves the backbone network. Finally, this paper uses all modules at the same time, that is, the model proposed in this paper has an accuracy improvement of 1.1% compared with the PEA+SC-DLA model, an accuracy improvement of 0.8% compared with the Cross Spatial Attention+SC-DLA model, and an accuracy improvement of 1.9% compared with the baseline. Through these five sets of experiments, we can conclude that the performance of target detection has been significantly improved by improving multi-level radar fusion, feature fusion method and backbone network.

[0039] Table 3

[0040]

Claims

1. A multi-level and multi-attention target detection method based on radar and camera, characterized in that: The steps include: Step 1: Obtain radar point clouds and images from public datasets and perform preprocessing; Step 2: construct the SC-DLA34 feature extraction network based on the DLA34 network to obtain the associated point cloud; Step 3: Multi-level fusion of the associated point cloud and the preprocessed image to obtain preliminary fusion features; Step 4: Concatenate the associated point cloud features with the preliminary fusion features at the channel level, and input the concatenated features into the PEA pyramid attention model to obtain deep fusion features. Step 5: Send the deep fusion features to the 3D detection head for decoding, estimate the depth, speed, rotation and attribute information of the detection target, and generate the target 3D detection frame.

2. The multi-level and multi-attention target detection method based on radar and camera according to claim 1 is characterized in that: The preprocessing includes the following operations: Perform random rotation data augmentation on the image; The radar point cloud is expanded using pillars to enhance its characterization capability: each radar point cloud is expanded into a pillar with dimensions of 0.2, 0.2, and 1.5 meters in the x, y, and z directions, respectively.

3. The multi-level and multi-attention target detection method based on radar and camera according to claim 2 is characterized in that: The specific implementation process of step 2 is as follows: The traditional pooling method in the downsampling process of the DLA34 network is changed to soft pooling, and the convolution in the process is replaced by a learnable and adaptive deformable convolution to construct the SC-DLA34 feature extraction network; Use SC-DLA34 as the feature extraction network to extract image features from the preprocessed image, and obtain the 2D detection frame of the target through the first-level feature detection head. The first-level detection head outputs three heads, namely, heat map features, center point position, and width and height of the detected target; The frustum association strategy proposed in Centerfusion is adopted to predict the 3D frustum of the region of interest using the 2D detection box and the depth information of the preprocessed point cloud. The point cloud data outside the region of interest is ignored to obtain the associated point cloud within the region of interest.

4. The multi-level and multi-attention target detection method based on radar and camera according to claim 3 is characterized in that: The soft pooling weights each local area of ​​the feature map by using the softmax function; the deformable convolution introduces a learnable offset into the traditional convolution operation to adjust the sampling position of the convolution kernel so that the convolution kernel fits the edge and shape of the target.

5. The multi-level and multi-attention target detection method based on radar and camera according to claim 4 is characterized in that: The specific implementation process of multi-level fusion of associated point clouds and images is as follows: The point cloud heat maps generated according to the target depth of the associated point cloud, the x-axis component and the y-axis component of the target radial velocity are used as the three channels of the associated point cloud features respectively; the associated point cloud features will be input into the multi-level fusion network with the image features extracted by the SC-DLA34 feature extraction network to output a preliminary fusion feature of the point cloud and image.

6. The multi-level and multi-attention target detection method based on radar and camera according to claim 5, characterized in that: The specific implementation process of the multi-level fusion network is as follows: A cross spatial attention model is proposed. The radar spatial attention weights are obtained by processing the associated point cloud features through the spatial attention network. The radar attention weights are used to guide the features obtained by the image features. After the multi-scale channel attention model MSCAM, channel attention is extracted through two branches with different scales. The local attention branch focuses on the scale problem of the channel through point-by-point convolution, and the global attention branch obtains the global context information through adaptive average pooling. The features processed by radar spatial attention and MSCAM are called new image features. The multi-level fusion network operation is repeated to output a preliminary fusion feature of point cloud and image.

7. The radar and camera-based multi-level multi-attention target detection method according to claim 6, characterized in that: The specific implementation process of the PEA pyramid attention model is as follows: The PEA pyramid attention model divides the feature channel into four parts, passes them through four deep convolutional network branches respectively, obtains four different scale features, and then concatenates these four scale features to obtain pyramid features. The pyramid features are directly passed through a convolutional layer after the global average pooling operation to obtain the attention weight, and finally the attention weight is multiplied by the feature tensor.

Images