A 3D Target Detection Method Based on 4D Millimeter-Wave Radar Point Clouds

Abstract

This invention discloses a 3D target detection method based on 4D millimeter-wave radar point clouds. It addresses the problems of sparse 4D millimeter-wave radar point clouds and lack of semantic information by employing a 4D millimeter-wave radar point cloud densification module and a multi-scale attention feature extraction module. The method includes densifying the input 4D millimeter-wave radar point cloud, averaging the features of points in voxels within the point cloud space, and then feeding the voxel points into a 3D backbone network for convolution processing to obtain 3D features at different scales and heights. Finally, after obtaining the BEV feature data, feature extraction is performed to predict the 3D bounding box position, size, classification, and orientation of the target object.

Description

Technical Field

[0001] This invention belongs to the field of autonomous driving perception, specifically relating to a 3D target detection method based on 4D millimeter-wave radar point clouds, and particularly to a 4D millimeter-wave radar point cloud feature extraction and target detection technology based on deep learning. Background Technology

[0002] In recent years, autonomous driving has received increasing attention due to its potential to reduce driver workload and improve driving safety. In modern autonomous driving systems, the perception system is an indispensable component, designed to accurately estimate the state of the surrounding environment and provide reliable observations for prediction and planning. The main task of 3D object detection in autonomous driving scenarios is to process input data from different sensors and output the category and precise location of the target object. Generally, the sensors used include LiDAR, millimeter-wave radar, and cameras.

[0003] Current research on 3D object detection algorithms mainly focuses on LiDAR and camera integration, primarily because LiDAR-generated point clouds can provide precise location and shape information of target objects. However, LiDAR is not a good solution due to its high cost and susceptibility to adverse weather conditions such as rain, fog, and strong light. Research on 2D object detection based on camera images is relatively mature. However, cameras only provide semantic features of a 2D plane, offering little help with the spatial information such as depth and orientation required for 3D object detection, and like LiDAR, it is greatly affected by light and weather. Traditional low-cost millimeter-wave Radar (x,y,v) is highly robust to adverse weather conditions and can provide crucial target velocity information in autonomous driving environments. However, limited by the radar technology of the time, traditional radar has low angular resolution and lacks height information, resulting in very sparse point clouds, thus it is rarely used for 3D object detection. The next generation of 4D Radar promises to solve this problem.

[0004] The four dimensions of 4D millimeter-wave radar are range, azimuth, elevation, and Doppler velocity, which can be converted to a commonly used 3D coordinate system (x, y, z, v) using formulas. The emergence of next-generation high-resolution 4D millimeter-wave imaging radar overcomes the shortcomings of traditional millimeter-wave radar, such as low resolution and lack of altitude information, and possesses point cloud output capabilities similar to lidar. Compared to lidar, 4D millimeter-wave radar has a mass production cost close to that of traditional millimeter-wave radar, is less affected by weather, and can achieve safe and high-speed operation in all weather conditions. Furthermore, utilizing the Doppler effect of millimeter-wave radar, it can accurately estimate the velocity and direction of targets. Therefore, in-depth research on this technology has significant research value and application prospects. Summary of the Invention

[0005] Purpose of the invention: This invention provides a target detection method based on 4D millimeter-wave radar, which uses a 4D millimeter-wave radar point cloud densification module and a multi-scale attention feature extraction module to solve the problems of the current 4D millimeter-wave radar point cloud being relatively sparse and lacking target semantic information.

[0006] Technical Solution: A 3D target detection method based on 4D millimeter-wave radar point clouds. The method extracts features from the 4D millimeter-wave radar point cloud using a 4D millimeter-wave radar point cloud densification module and a multi-scale attention feature extraction module, thereby regressing the 3D bounding box of the target object. The method includes the following steps:

[0007] (1) The input 4D millimeter-wave radar point cloud is densified, and each original radar point is used as a key point. Each key point has a corresponding three-dimensional position vector v. i and eigenvector f i For each key point, offset processing is performed, and then the key points and the original point cloud are added together to obtain dense 4D millimeter-wave radar point cloud data.

[0008] (2) Divide the entire point cloud space into L*W*H voxels using the point cloud data obtained in step (1), then perform feature averaging on the points in each voxel to obtain the features of each voxel, and then send the points in the voxel into the 3D backbone network.

[0009] (3) Based on three-dimensional sparse convolution, 3D backbone network is sampled for 3D sparse features. This includes sampling with a step size of 1 or 2 multiples based on SpatiallySparse Convolution. In the fusion processing of 3D sparse features, the feature sampling of voxels is divided into two or more branches for processing. Finally, the 3D sparse features under different branches are converted into 3D features under voxels and spliced ​​in the height dimension to obtain 3D features at different scales and heights.

[0010] (4) Based on the 3D features obtained in step (3), use the Squeeze and excitation operation in SEattention to extract global information and generate a weight vector. Then multiply the weight vector with the original 3D features in the height dimension, including enhancing some of the features, to obtain the enhanced 3D features.

[0011] (5) Compress the 3D features containing height into 2D BEV features. After obtaining the feature data of BEV, extract and stitch the features from the perspective of BEV and finally send them into the detection head.

[0012] (6) The detection head RPN network generates three branches to predict the position, size, classification, and orientation of the target object's 3D bounding box, respectively.

[0013] Furthermore, step (1) for processing key points includes the following specific steps:

[0014] (11) Perform offset calculations on the 3D position vector and feature vector of each key point to obtain a new feature vector f' i The calculation expression is as follows:

[0015]

[0016] Where ReLU is the activation function, W offset Let N(i) be a learnable weight matrix, and let N(i) be the neighboring point cloud of the i-th keypoint.

[0017] (12) Calculate the new key point position vector v' obtained after step (11). i :

[0018] v′ i =v i +tanh(W align [f′ i ])

[0019] Where tanh is the activation function, W align As a learnable weight matrix, each key point in the point cloud space is offset and added to the original point cloud to obtain dense 4D millimeter-wave radar point cloud data.

[0020] The three-dimensional sparse convolution mentioned in step (3) includes two methods: Spatially Sparse Convolution and Submanifold Sparse Convolution.

[0021] Furthermore, step (3) for the 3D backbone network includes downsampling the network to 1x, 2x, 4x, and 8x 3D sparse features using 3x3x3 three-dimensional sparse convolution, and then selecting features of three scales of 2x, 4x, and 8x for fusion.

[0022] Branch 1: First, perform a Spatially Sparse Convolution downsampling on the 2x features with a stride of 2, and then perform a sparse convolution on the features of Branch 1 with a stride of 1 to obtain sparse features with the same scale as 4x.

[0023] Branch 2: Use a single sparse convolution with a stride of 1 on the 4x features to obtain denser sparse features;

[0024] Branch 3: Use Sparse Inverse Convolution to upsample and obtain 3D sparse features at the same scale as Branch 1 and Branch 2. Then, convert the 3D sparse features obtained from the three branches into 3D features at voxel level and concatenate the three in the height dimension to obtain 3D features at different scales and heights.

[0025] Furthermore, step (5) specifically involves downsampling the feature map at different scales, then performing multiple convolutions on the feature maps at different scales to extract information, and then upsampling and splicing them in the channel dimension.

[0026] Beneficial effects: Compared with the prior art, the method provided by this invention first performs densification operation on 4D millimeter-wave radar point clouds, and then performs three-dimensional sparse convolution and compression to two-dimensional space for voxel feature extraction. It also enhances the sparse features in three-dimensional space. This not only improves the 3D target detection capability and quality of 4D millimeter-wave radar point clouds, but also greatly improves the accuracy and network model operation efficiency compared with the prior art, and has better application prospects and application significance. Attached Figure Description

[0027] Figure 1 This is a flowchart illustrating the implementation of the method described in this embodiment;

[0028] Figure 2 This is a diagram of the target network model structure implemented according to the present invention;

[0029] Figure 3 This is a schematic diagram of the coefficient point cloud densification module in the embodiment;

[0030] Figure 4 This is a schematic diagram of sparse convolution in this invention;

[0031] Figure 5 This is a schematic diagram of the target detection results in the embodiment. Detailed Implementation

[0032] To illustrate the technical solutions disclosed in this invention in detail, further description is provided below with reference to the accompanying drawings.

[0033] This invention provides a target detection method based on 4D millimeter-wave radar. Combining existing technology and the technical foundation known to those skilled in the art, the implementation steps of this invention are as follows:

[0034] Step 1: Perform densification operations on the input 4D millimeter-wave radar point cloud, such as... Figure 3 As shown, each original radar point is first taken as a key point (★ in the figure is used as one of the key points for illustration). Each key point has a corresponding three-dimensional position vector v.i and eigenvector f i Then, the new eigenvector f' is obtained using formula (1). i :

[0035]

[0036] Where ReLU is the activation function, W offset Let N(i) be a learnable weight matrix, and N(i) be the neighboring point cloud of the i-th keypoint (obtained using the ball lookup method, i.e., ...). Figure 3 Inside the middle ball Point cloud). Then, new key points are obtained using formula (2). Figure 3 The position vector ν' of the midpoint i :

[0037] ν′ i =v i +tanh(W align [f′ i ])

[0038] Where tanh is the activation function, W align This is a learnable weight matrix. Each keypoint in the point cloud space is offset and added to the original point cloud to obtain dense 4D millimeter-wave radar point cloud data.

[0039] Step 2: Using the point cloud data obtained in Step 1, divide the entire point cloud space into L*W*H voxels, then perform feature averaging on the points in each voxel to obtain the features of each voxel. Finally, feed the points in the voxels into the 3D backbone network.

[0040] Step 3: The network is downsampled to 1x, 2x, 4x, and 8x 3D sparse features using a series of 3x3x3 sparse convolutions. The main method used in the 3DBackbone module is 3D sparse convolution.

[0041] like Figure 4 The diagram illustrates two types of sparse convolution: Spatially Sparse Convolution and Submanifold Sparse Convolution. Spatially Sparse Convolution computes the convolution result as long as the kernel area covers a non-empty voxel. Submanifold Sparse Convolution computes the output only when the center of the kernel covers a non-empty voxel. In this case, the number of input voxels is the same as the number of output voxels.

[0042] Considering the accuracy of the balanced features and the computational efficiency of the network, such as Figure 2This invention fuses features at three scales: 2x, 4x, and 8x. Branch 1 first performs a Spatially Sparse Convolution downsampling of 2 on the 2x features, and then performs a sparse convolution of 1 on the features of this branch to obtain sparse features at the same scale as 4x.

[0043] It should be noted that because the point cloud of 4Dradar is very sparse, using Spatially Sparse Convolution twice consecutively not only obtains a larger receptive field but also yields denser sparse features. For the 4x features in branch two, this invention uses only one sparse convolution with a stride of 1 to obtain denser sparse features. Branch three uses Sparse Inverse Convolution for upsampling to obtain 3D sparse features at the same scale as branches one and two. Then, the 3D sparse features obtained from the three branches are converted into 3D features at voxel level, and the three are concatenated along the height dimension. This results in 3D features at different scales and heights.

[0044] Current 3D object detection algorithms typically compress 3D features containing height into 2D BEV (Browser-Edge-Variant) features. This is mainly because in road scenes, almost no objects overlap in height, and BEVs represent all objects with virtually no vertical occlusion. However, this height compression process results in the loss of important height information for some objects. Therefore, inspired by SENet and EPSANet, a multi-scale attention module was designed to enhance 3D features at different scales and heights, reducing feature loss during height compression.

[0045] Step 4, as follows Figure 2 As shown, this module first uses the Squeeze and excitation operations in SE attention to extract global information and generate a weight vector. Then, the weight vector is multiplied with the original 3D features in the height dimension to enhance the features at more important heights, resulting in enhanced 3D features.

[0046] Step 5: After obtaining the feature data of the BEV, feature extraction is performed on the feature map from the perspective of the BEV. First, the feature map is downsampled at different scales. Then, multiple convolutions are performed at different scales to extract information. Finally, upsampling is performed, and the data is concatenated along the channel dimension. The data is then fed into the detection head.

[0047] Step 6: The detection head RPN network generates three branches to predict the position, size, classification, and orientation of the target object's 3D bounding box, respectively.

[0048] Experiments have shown that the method of this invention has excellent target detection capabilities, such as... Figure 5 As shown, it can accurately predict the location, size, and classification information of the target vehicle.

Claims

1. A 3D target detection method based on 4D millimeter-wave radar point clouds, characterized in that: The method extracts 4D millimeter-wave radar point cloud features through a 4D millimeter-wave radar point cloud densification module and a multi-scale attention feature extraction module to regress the 3D bounding box of the target object, including the following steps: (1) The input 4D millimeter-wave radar point cloud is densified, and each original radar point is used as a key point. Each key point has a corresponding three-dimensional position vector v. i and eigenvector f i For each key point, offset processing is performed, and then the key points and the original point cloud are added together to obtain dense 4D millimeter-wave radar point cloud data. (2) Divide the entire point cloud space into L*W*H voxels using the point cloud data obtained in step (1), then perform feature averaging on the points in each voxel to obtain the features of each voxel, and then send the points in the voxel into the 3D backbone network. (3) Based on three-dimensional sparse convolution, 3D backbone network is sampled for 3D sparse features. This includes sampling with a step size of 1 or 2 multiples based on SpatiallySparse Convolution. In the fusion processing of 3D sparse features, the feature sampling of voxels is divided into two or more branches for processing. Finally, the 3D sparse features under different branches are converted into 3D features under voxels and spliced ​​in the height dimension to obtain 3D features at different scales and heights. (4) Based on the 3D features obtained in step (3), use the Squeeze and excitation operation in SE attention to extract global information and generate a weight vector. Then multiply the weight vector with the original 3D features in the height dimension, including enhancing some of the features, to obtain the enhanced 3D features. (5) Compress the 3D features containing height into 2D BEV features. After obtaining the feature data of BEV, extract and stitch the features from the perspective of BEV and finally send them into the detection head. (6) The detection head RPN network generates three branches to predict the position, size, classification, and orientation of the target object's 3D bounding box, respectively.

2. The 3D target detection method based on 4D millimeter-wave radar point clouds according to claim 1, characterized in that: Step (1) for processing key points includes the following specific steps: (11) Perform offset calculation on the three-dimensional position vector and feature vector of each key point to obtain a new feature vector f′. i The calculation expression is as follows: Where ReLU is the activation function, W offset Let N(i) be a learnable weight matrix, and let N(i) be the neighboring point cloud of the i-th keypoint. (12) Calculate the new key point position vector ν' obtained after step (11). i : v′ i =v i +tanh(W align [f′ i ]) Where tanh is the activation function, W align As a learnable weight matrix, each key point in the point cloud space is offset and added to the original point cloud to obtain dense 4D millimeter-wave radar point cloud data.

3. The 3D target detection method based on 4D millimeter-wave radar point clouds according to claim 1, characterized in that: The three-dimensional sparse convolution mentioned in step (3) includes two methods: Spatially Sparse Convolution and Submanifold Sparse Convolution.

4. The 3D target detection method based on 4D millimeter-wave radar point clouds according to claim 3, characterized in that: For the 3D backbone network, a 3x3x3 three-dimensional sparse convolution is used to downsample the network to 1x, 2x, 4x, and 8x 3D sparse features, and then the features at the three scales of 2x, 4x, and 8x are fused. Branch 1: First, perform a Spatially Sparse Convolution downsampling on the 2x features with a stride of 2, and then perform a sparse convolution on the features of Branch 1 with a stride of 1 to obtain sparse features with the same scale as 4x. Branch 2: Use a single sparse convolution with a stride of 1 on the 4x features to obtain denser sparse features; Branch 3: Use Sparse Inverse Convolution to upsample and obtain 3D sparse features at the same scale as Branch 1 and Branch 2. Then, convert the 3D sparse features obtained from the three branches into 3D features at voxel level and concatenate the three in the height dimension to obtain 3D features at different scales and heights.

5. The 3D target detection method based on 4D millimeter-wave radar point clouds according to claim 1, characterized in that: Step (5) specifically involves downsampling the feature map at different scales, then performing multiple convolutions at different scales to extract information, and finally upsampling and then splicing them in the channel dimension.

Images