3D target detection method based on fusion of camera and enhanced 4D millimeter wave radar

Abstract

According to the 3D target detection method based on fusion of the camera and the enhanced 4D millimeter-wave radar, a simple and effective point cloud enhancement algorithm is designed, and the sparsity problem of point clouds is relieved by adding effective four-dimensional radar point clouds; a point cloud feature fusion enhancement module is developed, target detail features are enhanced by constructing sparse voxels of different scales, and the system robustness is improved.

Description

Technical Field

[0001] The present invention belongs to the field of autonomous driving technology, specifically to achieve the deep integration of artificial intelligence and people-oriented autonomous driving under Industry 5.0, and is a training method for integrating multimodal data to better complete detection tasks. Background Art

[0002] With the rise of autonomous driving technology, breakthroughs have been made in the research of perception systems based on cameras, millimeter-wave radars, and lidars, providing sufficient information for subsequent trajectory prediction, motion planning, and control. These systems have been successful in well-defined and predictable urban scenarios. However, 3D object detection, as the most widely studied area in autonomous driving perception, still faces severe challenges in severe weather and dynamic off-road scenarios. In severe weather conditions, the attenuation of lidar increases significantly, which affects its propagation distance and may cause fog or haze to be mistaken for obstacles. In addition, due to the low scanning frequency and narrow laser beam of lidar, spatial target search becomes very difficult, limiting the scope of target search and capture. Cameras are sensitive to light and weather conditions; insufficient light, strong light, shadows, rain and snow can affect image quality and reduce camera performance and reliability. 4D millimeter-wave (mm-Wave) radar has attracted much attention due to its all-weather and all-weather sensing capabilities and its strong robustness under severe weather conditions and target occlusion. Its combination with cameras can provide high-resolution visual information and capture rich semantic features, and is a current research hotspot. Although the fusion of cameras and 4D millimeter-wave radars can solve the perception challenges in severe weather and off-road scenarios, the sparsity and disorder of 4D millimeter-wave radar point clouds affect the detection accuracy. Summary of the invention

[0003] In view of this, the present invention proposes a 3D target detection method that integrates a camera and an enhanced 4D millimeter-wave radar, which improves the training effect and prediction accuracy of the model by expanding the sparse 4D millimeter-wave radar point cloud.

[0004] A 3D target detection method integrating a camera and an enhanced 4D millimeter wave radar, comprising:

[0005] Step 1: Expand the sparse radar point cloud data;

[0006] Step 2: voxelize the expanded point cloud using large and small sizes; then add the point cloud features under the two sizes of voxels to obtain radar point cloud feature data;

[0007] Step 3: Fusing the image feature data obtained by the camera with the radar point cloud feature data obtained in step 2;

[0008] Step 4: Based on the fused data in step 3, use the detection head to detect the target.

[0009] Preferably, in step 1, the method for expanding the point cloud of sparse radar point cloud data includes:

[0010] After obtaining the 4D millimeter-wave radar points, a set number of points are randomly generated within the current range of each point cloud. In the process of determining the features of the generated points, the nearest K point clouds around each randomly generated point are found, and the feature mean of these adjacent points is calculated as the new feature of the generated point.

[0011] Preferably, in step 2, for any point cloud point P of each size voxel, after voxel feature extraction, its ten-dimensional information is expressed as: P = [x, y, z, RCS, x c ,y c ,z c ,x v ,y v ,z v ], where (x, y, z) represents the spatial coordinates of the point cloud point in the radar coordinate system; RCS is the reflectivity; (x c ,y c ,z c ) represents the difference between the point cloud coordinates and the voxel center coordinates; (x v ,y v ,z v ) represents the difference between the point cloud coordinates and the coordinates of the point cloud cluster center inside the voxel; finally, the point cloud point features of the two sizes of voxels are encoded to obtain the radar point cloud feature data.

[0012] Preferably, in step 2, the voxel sizes of the two sizes are 8 times of each other.

[0013] Preferably, in step 3, the image feature data obtained by the camera is fused with the radar point cloud feature data obtained in step 2 using the MVXnet fusion method.

[0014] Preferably, in step 4, based on the data fused in step 3, the method of using the detection head to detect the target includes:

[0015] The ground truth bounding box and anchor point are represented as (x, y, z, w, l, h, θ); where w and l represent the width and length of the bounding box, respectively; h and θ represent the height and heading angle of the target, respectively; the difference between the ground truth bounding box and the predicted anchor point is calculated as follows:

[0016]

[0017] Δθ=sin(θ gt -θ a )

[0018] Among them, (x a ,y a , z a ) represents the coordinates output by the detection head; w a and l a Respectively represent the width and length of the bounding box output by the detection head; (x gt ,y gt , z gt ) represents the coordinate value of the ground truth bounding box; w gt and h gt denote the width and height of the ground truth bounding box respectively. gt ,θ a Respectively represent the true value and predicted value of yaw around the z-axis;

[0019] d a =((w a ) 2 +(l a ) 2 ) (1 / 2) Represents the diagonal length of the anchor bottom. The loss of the bounding box is calculated as follows:

[0020]

[0021] Using the same binary cross entropy loss L cls and direction loss L dir , the total loss is expressed as follows:

[0022]

[0023] Among them, N pos represents the number of positive anchor points, β 1 =2,β 2 =1,β 3 =0.2.

[0024] The present invention has the following beneficial effects:

[0025] The present invention proposes a 3D target detection method by fusing a camera with an enhanced 4D millimeter-wave radar, designs a simple and effective point cloud enhancement algorithm, alleviates the sparsity problem of point cloud by adding effective four-dimensional radar point cloud; develops a point cloud feature fusion enhancement module, enhances target detail features by constructing sparse voxels of different scales, and improves system robustness. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 Augmenting model algorithm architecture for point clouds.

[0027] Figure 2 This is the algorithm architecture of the dual-voxel feature fusion enhancement module.

[0028] Figure 3 This is the system framework of the complex road infrastructure model proposed in this invention.

[0029] Figure 4 A visualization diagram of the model detection results is proposed for the present invention. DETAILED DESCRIPTION

[0030] 1. Point cloud enhancement algorithm based on KNN:

[0031] like Figure 1 As shown in the figure, the present invention proposes a point cloud expansion model (PCFE) based on sparse radar point cloud data. Due to its inherent sparsity, 4D millimeter-wave radar point cloud data often cannot provide sufficient point cloud density to support high-precision model training in practical applications. In order to solve this problem, after obtaining the 4D millimeter-wave radar points, we randomly generate 40 points within the current range of the point cloud. The generated points are consistent with the initial 4D millimeter-wave radar point cloud features and have the following characteristics:

[0032] P = [x, y, z, RCS]

[0033] Among them, x, y, z are the spatial coordinates in the radar coordinate system, and RCS is the reflectivity. In order to ensure that these randomly generated points can participate in model training as valid data, rather than becoming noise data that affects detection accuracy, the K-nearest search method is introduced. In the process of determining the features of the generated points, the nearest K point clouds around each randomly generated point are found, and the feature mean of these neighboring points is calculated and used as the new feature of the generated point. The new feature can be expressed as:

[0034]

[0035] Among them, N is the number of adjacent point clouds, and Pi represents the characteristics of the i-th point cloud. The purpose of this is to ensure that the newly generated points can accurately reflect the characteristics of the surrounding point clouds, thereby improving the validity of the data. It is worth noting that due to the sparsity of 4D millimeter-wave radar point clouds, it is necessary to be extremely cautious when selecting the K value. If the K value is set too high, the generated points may capture the characteristics of point clouds at a longer distance, thereby deviating from the actual characteristics of their location. This will not only fail to improve the data quality, but will introduce noise and affect the performance of the model. Through a series of experimental comparative analysis, it was finally determined to set the K value to 3. This choice can maximize the use of effective surrounding information while ensuring feature accuracy.

[0036] After the above expansion processing of the sparse 4D millimeter wave radar point cloud, the density and uniformity of the point cloud have been significantly enhanced. This not only improves the quality of the data, but also improves the performance of the model in the training phase, thereby achieving better results in prediction accuracy.

[0037] 2. Dual-voxel feature fusion enhancement algorithm

[0038] Considering the use of VoxelNet for voxelization, points and voxels that exceed the buffer capacity will be discarded, and filling voxels will increase the computational load. This time, dynamic voxelization is used to avoid sampling fixed points or voxel capacity to form a buffer, but to maintain a one-to-one mapping between points and voxels. Therefore, the number of voxels is adjusted as the number of point clouds changes. During the voxelization process, it can be observed that the voxel size significantly affects the accuracy and robustness of the model. Specifically, a smaller voxel size may capture noise in the data, reduce the robustness of the model and may cause overfitting. On the other hand, a larger voxel size may not capture fine local features, resulting in loss of details. On this basis, a dual voxel feature fusion enhancement module (DVFFE) is proposed, such as Figure 2 In this module, the voxelization of the point cloud is divided into two independent voxel sizes. Branch A uses relatively small voxels to capture detailed features, while branch B uses larger voxels to reduce data noise, where the voxel size of A is 1 / 8 of the voxel size of B. Each voxel branch extracts its voxel features independently.

[0039] For point P, after voxel feature extraction, its ten-dimensional information is expressed as: P = [x, y, z, RCS, x c ,y c ,z c ,x v ,y v ,z v ], where xyz and RCS represent the basic information of the point cloud, and the subscripts c and v represent the difference between the point cloud coordinates and the voxel center coordinates and the point cloud cluster center coordinates inside the voxel, respectively. In addition, the voxel feature encoding (VFE) layer encodes the features of the dual-branch voxels. The dual-branch voxels are mapped to a three-dimensional coordinate system for feature alignment. Subsequently, the dual-branch voxel features are added.

[0040] 3. Fusion of the image data obtained by the camera and the millimeter-wave radar fusion feature data obtained in step 2

[0041] Based on the feature information obtained from both modalities, the present invention uses the fusion method of MVXnet to merge them. First, the 3D point cloud is projected onto the image using a projection matrix. Secondly, based on the index obtained from the projection, the corresponding image features are attached to the point cloud and applied to the image feature map at all levels. In this process, information about the existence of the object is added; if an object is detected, its posture is associated from the 2D image to each 3D point. Finally, the information from the two modes is concatenated. This method integrates information from both modes early on, allowing the network to learn and integrate useful information from both. Figure 3 The overall structure of the model is shown.

[0042] 4. Target Detection Based on Detection Head

[0043] We use an anchor-based detection head and represent the ground truth bounding box and anchor as (x, y, z, w, l, h, θ), where w and l represent the width and length of the bounding box, respectively; h and θ represent the height and heading angle of the object, respectively.

[0044] The difference between the ground truth bounding box and the predicted anchor point is calculated as follows:

[0045]

[0046] Δθ=sin(θ gt -θ a )

[0047] Among them, (x a ,y a , z a ) represents the coordinates output by the detection head; w a and l a Respectively represent the width and length of the bounding box output by the detection head; (x gt ,y gt , z gt ) represents the coordinate value of the ground truth bounding box; w gt and h gt denote the width and height of the ground truth bounding box respectively. gt ,θ a Respectively represent the true value and predicted value of yaw around the z-axis;

[0048] d a =((w a ) 2 +(l a ) 2 ) (1 / 2) Represents the diagonal length of the anchor bottom. The loss of the bounding box is calculated as follows:

[0049]

[0050] We use the same binary cross entropy loss L cls and direction loss L dir The total loss is expressed as follows:

[0051]

[0052] Among them, N pos represents the number of positive anchor points, β 1 =2,β 2 =1,β 3 =0.2.

[0053] As shown in Table 1, we demonstrate the effectiveness of the proposed model in experiments and our model achieves better evaluation indicators compared with other models.

[0054] Table 1 Detection results based on VOD dataset

[0055]

[0056] Meanwhile, a comprehensive ablation study of the proposed algorithm is performed, as shown in Table 2. We observe that the introduction of the DVFFE module significantly improves the baseline, demonstrating its ability to effectively reduce noise while extracting detailed features. The introduction of the PCFE algorithm increases the number of valid virtual points. Combining all modules together produces significant improvements, confirming the effectiveness of our design.

[0057] Table 2 ERC-Fusion model ablation experiment

[0058]

[0059]

[0060] In summary, the above are only preferred embodiments of the present invention and are not intended to limit the protection scope of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A 3D target detection method integrating a camera and an enhanced 4D millimeter wave radar, characterized in that: include: Step 1: Expand the sparse radar point cloud data; Step 2: voxelize the expanded point cloud using large and small sizes; then add the point cloud features under the two sizes of voxels to obtain radar point cloud feature data; Step 3: Fusing the image feature data obtained by the camera with the radar point cloud feature data obtained in step 2; Step 4: Based on the fused data in step 3, use the detection head to detect the target.

2. The 3D target detection method of the camera and enhanced 4D millimeter wave radar fusion as claimed in claim 1, characterized in that: In step 1, the method for expanding the point cloud of sparse radar point cloud data includes: After obtaining the 4D millimeter-wave radar points, a set number of points are randomly generated within the current range of each point cloud. In the process of determining the features of the generated points, the nearest K point clouds around each randomly generated point are found, and the feature mean of these adjacent points is calculated as the new feature of the generated point.

3. The 3D target detection method of the camera and enhanced 4D millimeter wave radar fusion as claimed in claim 1 or 2, characterized in that: In step 2, for any point cloud point P of each size voxel, after voxel feature extraction, its ten-dimensional information is expressed as: P = [x, y, z, RCS, x c ,y c ,z c ,x v ,y v ,z v ], where (x, y, z) represents the spatial coordinates of the point cloud point in the radar coordinate system; RCS is the reflectivity; (x c ,y c ,z c ) represents the difference between the point cloud coordinates and the voxel center coordinates; (x v ,y v ,z v ) represents the difference between the point cloud coordinates and the coordinates of the point cloud cluster center inside the voxel; finally, the point cloud point features of the two sizes of voxels are encoded to obtain the radar point cloud feature data.

4. The 3D target detection method of the camera and enhanced 4D millimeter wave radar fusion as claimed in claim 3, characterized in that: In step 2, the voxel sizes of the two sizes are 8 times the size of each other.

5. The 3D target detection method of the camera and enhanced 4D millimeter wave radar fusion as claimed in claim 1, characterized in that: In step 3, the image feature data obtained by the camera is fused with the radar point cloud feature data obtained in step 2 using the MVXnet fusion method.

6. The 3D target detection method of the camera and enhanced 4D millimeter wave radar fusion as claimed in claim 1, characterized in that: In step 4, based on the data fused in step 3, the method of detecting the target using the detection head includes: The ground truth bounding box and anchor point are represented as (x, y, z, w, l, h, θ); where w and l represent the width and length of the bounding box, respectively; h and θ represent the height and heading angle of the target, respectively; the difference between the ground truth bounding box and the predicted anchor point is calculated as follows: Δθ=sin(θ gt -θ a ) Among them, (x a ,y a , z a ) represents the coordinates output by the detection head; w a and l a Respectively represent the width and length of the bounding box output by the detection head; (x gt ,y gt , z gt ) represents the coordinate value of the ground truth bounding box; w gt and h gt denote the width and height of the ground truth bounding box respectively; θ gt ,θ a Respectively represent the true value and predicted value of yaw around the z-axis; d a =((w a ) 2 +(l a ) 2 ) (1 / 2) represents the diagonal length of the anchor bottom; the loss of the bounding box is calculated as follows: Using the same binary cross entropy loss L cls and direction loss L dir , the total loss is expressed as follows: Among them, N pos Represents the number of positive anchor points, β1=2, β2=1, β3=0.2.

Images