3D target detection method based on 4D millimeter wave point cloud and image cross-view fusion

Abstract

The invention discloses a 3D target detection method based on 4D millimeter wave point cloud and image cross-view fusion. The invention provides a two-stage cross view fusion network for 3D target detection of a 4D millimeter wave radar and a camera by utilizing relatively accurate space positioning capability of 4D millimeter waves and semantic information of images. The first stage radar-guided BEV multi-scale fusion provides a high recall proposal box. In the second stage, a point cloud guide fusion module and a grid guide fusion module are designed, and various forms of features including point cloud, forward view images and bird's-eye view (BEV) can be effectively utilized. The cross-view-angle features are subjected to cross fusion to obtain proposal box features at an instance level, and fine help is provided for the refinement of the proposal box. Experimental results show that compared with an existing method, the network designed by the invention obtains better 3D detection precision, and the target detection capability in a complex environment is improved.

Description

Technical Field

[0001] The present invention relates to a 3D object detection method in the field of radar point cloud fusion image perception, and particularly to a 3D object detection method for cross-view fusion of 4D millimeter-wave radar point cloud and image. Background Art

[0002] 3D object detection is an important perception task for autonomous vehicles. Millimeter-wave radar has received extensive attention due to its robustness in adverse environments such as rainy days, foggy and snowy weather. Compared with traditional 3D radars that only provide azimuth, distance, and Doppler velocity measurements, 4D radars can obtain additional valuable height measurements, generating a 3D point cloud similar to lidar. However, compared with lidar, the point cloud of 4D radar is very sparse, usually with 0.1k - 2k points per frame. In addition, due to the multipath effect of millimeter waves, radar measurements are much noisier than lidar. Therefore, relying solely on 4D radar for three-dimensional object detection is not very satisfactory. Most current research focuses on fusing the 4D radar with the images of cameras to improve the detection performance.

[0003] Current 4D radar and camera fusion work generally adopts a one-stage paradigm, where the features of the two modalities are respectively transformed and fused in a unified BEV space. Although simple and effective, this pipeline has a feature misalignment problem. Due to the depth ambiguity of monocular images, position drift occurs when converting the front view of image features to the BEV view. At the same time, different from lidar, sparse and noisy radar points cannot provide sufficient and accurate 3D information for the image. Therefore, directly stitching the BEV features of radar and camera will inevitably lead to misalignment, especially for small objects, resulting in large prediction deviations or false positives. Summary of the Invention

[0004] To solve the problems existing in the background art, the present invention proposes a 3D object detection method for 4D millimeter-wave radar point cloud and image cross-view fusion. The present invention first generates some proposal boxes, and then extracts the features in the proposal boxes for refinement. In this way, the network consists of two stages closely and can gradually refine and align the features through various fusion mechanisms. To improve the recall rate of the proposal boxes, the present invention introduces a radar-guided BEV multi-scale fusion module, which can achieve better feature alignment in the first stage. In the second stage, the present invention realizes cross-view fusion to refine the proposal boxes. To fully explore the features of different views from heterogeneous modalities, the present invention designs two modules in the second stage for deep fusion of features, namely the point cloud-guided fusion module and the grid-guided fusion module. The point cloud-guided fusion module is responsible for retrieving the radar points near the proposal boxes and fusing the image features corresponding to the point clouds. Considering that some proposal boxes may not be assigned with valuable features due to the lack of radar points, the present invention further designs a grid-guided fusion module, which can allocate and fuse features according to the spatial grids within the proposal boxes, and sequentially aggregate the front-view image features and the BEV features. Due to the spatial orthogonality of the image front view and the BEV, the bounding boxes of the proposal boxes can be further refined in the three-dimensional space of the two perspectives. The method of the present invention has been extensively tested on the public datasets VoD and TJ4DRadSet, and significant performance improvements have been achieved compared with the state-of-the-art methods.

[0005] The technical solution adopted by the present invention is as follows:

[0006] I. A 3D object detection method for 4D millimeter-wave point cloud and image cross-view fusion

[0007] 1) Use a 3D backbone network to extract voxel features and BEV features from the 4D millimeter-wave radar point cloud, and obtain the corresponding multi-scale radar voxel features and multi-scale radar BEV features; and generate the corresponding image 2D features and image BEV features of the image;

[0008] 2) Use the radar-guided BEV multi-scale fusion module to perform feature fusion on the radar BEV features and the image BEV features to obtain BEV fusion features; and then predict 3D proposal boxes based on the BEV fusion features;

[0009] 3) The point cloud-guided fusion module performs point cloud-guided feature fusion according to each 3D proposal box, the radar voxel features, and the image 2D features, and obtains the proposal box features enhanced by the first-view features;

[0010] 4) The grid-guided fusion module performs grid-guided feature fusion according to each 3D proposal box, the image 2D features, and the radar BEV features, and obtains the proposal box features enhanced by the second-view features;

[0011] 5) Concatenate the proposal box features enhanced by the two view features corresponding to each 3D proposal box and perform self-attention fusion to obtain the final proposal box features. After the deep learning neural network regresses the offsets based on the final proposal box features and refines and corrects the proposal boxes based on the obtained offsets, the final 3D object detection boxes are obtained.

[0012] The extraction of voxel features and BEV features from the 4D millimeter-wave radar point cloud using the 3D backbone network to obtain the corresponding multi-scale radar voxel features and radar BEV features is specifically as follows:

[0013] First, after performing voxel division on the 4D millimeter-wave radar point cloud, the radar point cloud after voxel division is obtained, and then the SECOND network is used to extract the corresponding multi-scale radar voxel features and BEV features of the radar point cloud after voxel division.

[0014] The generation of the image 2D features and image BEV features corresponding to the image is specifically as follows:

[0015] Use the 2D backbone network to extract 2D features of the image, and then perform view transformation on the extracted image 2D features to obtain the image BEV features.

[0016] In the above (2), the radar-guided BEV multi-scale fusion module is used to fuse the radar BEV features and the image BEV features to obtain the BEV fusion features, specifically as follows:

[0017] The BEV fusion features are obtained by fusing the multi-scale BEV fusion features, and the generation steps of each scale of BEV fusion features are as follows:

[0018] Predict the BEV occupancy probability map of the current scale based on each scale of radar BEV features; then use the occupancy probability map to enhance the image BEV features to obtain the enhanced image BEV features; then, fuse the enhanced image BEV features with the radar BEV features of the current scale to obtain the BEV fusion features of the current scale.

[0019] In the above (2), 3D proposal boxes are predicted based on the BEV fusion features, specifically as follows:

[0020] Input the BEV fusion features into the proposal box generation network, and the network outputs several 3D proposal boxes.

[0021] The above (3) is specifically as follows:

[0022] The point cloud-guided fusion module includes a connected cross-modal deformable attention mechanism and region of interest pooling; first, the radar voxel features and the image 2D features are fused through the cross-modal deformable attention mechanism to obtain the point cloud fusion features with enhanced image features Then, the image features near the current 3D proposal box are aggregated through region of interest pooling to enhance the fused point cloud features. Obtain the proposal box features after the first view feature enhancement.

[0023] The specific content of the 4) is as follows:

[0024] First, each 3D proposal box is divided into regular grids and the grids are feature encoded to obtain the grid features of the current 3D proposal box; then, the grid features of the current 3D proposal box, the image 2D features and the radar BEV features are used as the input of the first cross-modal deformable attention mechanism, the first cross-modal deformable attention mechanism is connected to the second cross-modal deformable attention mechanism, and the second cross-modal deformable attention mechanism outputs the proposal box features after the second view feature enhancement.

[0025] II. A computer device

[0026] The device includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, the steps of the 3D object detection method for 4D millimeter-wave point cloud and image cross-view fusion are implemented.

[0027] III. A computer-readable storage medium

[0028] A computer program is stored on the medium, and when the computer program is executed by a processor, the steps of the 3D object detection method for 4D millimeter-wave point cloud and image cross-view fusion are implemented.

[0029] IV. A computer program product

[0030] The product includes computer programs / instructions, and when the computer programs / instructions are executed by a processor, the steps of the 3D object detection method for 4D millimeter-wave point cloud and image cross-view fusion are implemented.

[0031] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0032] Compared with the existing one-stage detection paradigm, the present invention proposes a new two-stage cross-view fusion network for 3D object detection of 4D radar and camera. The radar-guided multi-scale BEV fusion provides proposal boxes with high recall rate. In the second stage, the fusion based on the point-guided fusion component and the grid-guided fusion component utilizes various forms of features, including points, front-view images and BEV, providing comprehensive clues for the refinement of the proposal boxes. Experimental results show that the network of the present invention is much better than the state-of-the-art methods, revealing the great potential of 4D radar and camera fusion in autonomous driving. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1Shows the flowchart of the present invention.

[0034] Figure 2 Is the structural diagram of the radar-guided BEV multi-scale fusion module.

[0035] Figure 3 Is the structural diagram of the cross-modal deformable attention mechanism.

[0036] Figure 4 Is the visualization comparison chart of the detection results of the present invention and the LoGoNet method. Detailed implementation manners

[0037] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0038] The present invention proposes a 3D object detection method for cross-view fusion of 4D millimeter-wave point clouds and images. As Figure 1 shown, the method includes the following steps:

[0039] 1) Use a 3D backbone network to extract voxel features and BEV features from the 4D millimeter-wave radar point cloud, obtaining corresponding multi-scale radar voxel features and multi-scale radar BEV features; and generate image 2D features and image BEV features corresponding to the image;

[0040] Among them, using a 3D backbone network to extract voxel features and BEV features from the 4D millimeter-wave radar point cloud, obtaining corresponding multi-scale radar voxel features and radar BEV features, specifically:

[0041] First, after voxel partitioning the 4D millimeter-wave radar point cloud, obtain the radar point cloud after voxel partitioning, and then use the SECOND network to extract the corresponding multi-scale radar voxel features and BEV features of the radar point cloud after voxel partitioning.

[0042] A frame of millimeter-wave point cloud P can be expressed as where (x i , y i , z i ) is the spatial coordinate of the i-th point, is the feature including the Doppler velocity and radar cross section (RCS) of each point, etc., and N is the number of point clouds. After multiple layers of 3D sparse convolution in SECOND, multi-scale radar voxel features can be obtained, where is the resolution of the voxel, and C V,i is the voxel feature channel number. Select the voxel features with i = 2, 3, 4 and convert them into BEV features where C B,k is the BEV feature channel number, and X, Y, Z are the numbers of voxels in the x, y, z three directions respectively.

[0043] Generate the 2D image features and BEV image features corresponding to the image, specifically as follows:

[0044] Use a 2D backbone network to extract 2D features of the image, and then perform view transformation on the extracted 2D image features to obtain BEV image features. In this embodiment, the 2D backbone network is Swin-Transformer, and the 2D image features where W I , H I are the width and height of the image; C I is the number of channels of the image features. For the extracted 2D image features F I Performing view transformation to obtain BEV image features specifically adopts a depth-based view transformation method to project the 2D image features onto the BEV plane, thereby obtaining BEV image features where C B is the number of channels of the camera BEV features.

[0045] 2) Use the radar-guided BEV multi-scale fusion module to perform feature fusion on the radar BEV features and the image BEV features to obtain BEV fusion features; then predict 3D proposal boxes based on the BEV fusion features;

[0046] Among them, using the radar-guided BEV multi-scale fusion module to perform feature fusion on the radar BEV features and the image BEV features to obtain BEV fusion features, specifically as follows:

[0047] The BEV fusion features are obtained by fusing the multi-scale BEV fusion features. Specifically, the BEV features of three scales are fused by up and down aggregating to the scale of k = 1 as the final BEV features:

[0048]

[0049] Among them, Down and Up respectively represent reducing and doubling the resolution of the feature map by bilinear interpolation.

[0050] The generation steps of the BEV fusion features at each scale are as follows:

[0051] According to the radar BEV features at each scale k Predict the BEV occupancy probability map W k at the current scale. Specifically, as Figure 2 shown, at each scale k, the present invention first generates a weight map W k representing the occupancy probability of the BEV grid from the corresponding radar BEV features through the convolutional layer Conv2d and the Sigmoid function, as follows:

[0052]

[0053] Re - utilize the occupancy probability map to enhance the image BEV features, obtaining the enhanced image BEV features As Figure 2 shown, specifically, multiply the BEV occupancy probability map W k by the corresponding image BEV feature map to enhance the position information of the image BEV features:

[0054]

[0055] where ⊙ represents the multiplication operation.

[0056] After obtaining the image BEV features at scale k, pass them through a convolutional layer with a stride of 2 to obtain the image BEV features at scale k + 1 for subsequent iterative fusion:

[0057]

[0058] Then, fuse the enhanced image BEV features with the radar BEV features at the current scale to obtain the BEV fusion features at the current scale. Specifically, concatenate the radar BEV features and the image BEV features at the same scale k and then perform 2D convolutional fusion:

[0059]

[0060] Based on the BEV fusion features, predict 3D proposal boxes, specifically:

[0061] Input the BEV fusion features into a proposal box generation network composed of a classification head and a regression head, and the network outputs several 3D proposal boxes b1, b2, …, b n and form a proposal box set B = {b1, b2, …, b n}.

[0062] 3) The point - cloud - guided fusion module performs point - cloud - guided feature fusion based on each 3D proposal box, radar voxel features, and image 2D features to obtain the proposal box features with enhanced first - view features;

[0063] 3) Specifically:

[0064] Radar point clouds have strong spatial prior information, while images have dense semantic information. The point cloud-guided fusion module aims to fuse radar and image features in the form of points. The point cloud-guided fusion module includes a connected cross-modal deformable attention mechanism and region of interest pooling (RoI-Pooling); first, the radar voxel features and image features are fused through the cross-modal deformable attention mechanism to obtain the point cloud fusion features with enhanced image features The proposed bounding boxes are divided into a U*U*U uniform grid, and each grid aggregates the point cloud fusion features within a radius R centered on the grid l inside the sphere Then, the point cloud fusion features with enhanced image features near the current 3D proposed bounding box are aggregated through region of interest pooling The proposed bounding box features with enhanced first-view features are obtained. Region of interest pooling uses max pooling for all points within the grid, and finally combines all pooled grid features into the proposed bounding box features where b and C b represent the number of proposed bounding boxes and the number of feature channels respectively represents the number of grids

[0065] Among them, the structural diagram of the cross-modal deformable attention mechanism (CMDA) is as shown in Figure 3 First, each radar point p in the given 3D proposed bounding box is retrieved, and the corresponding projection point r in the image plane can be obtained by multiplying with the projection matrix of the camera p , satisfying r p =(u p ,v p ), where u p ,v p are the two-dimensional coordinates of r p on the image coordinate system, and the formula is as follows

[0066] r p (u p ,v p ) = T·p(x,y,z)

[0067] where (u,v) is the projected pixel position on the image plane, T is the projection matrix of the camera, and p(x,y,z) is the three-dimensional coordinate of the radar point p

[0068] Then, the radar point feature f p is used as the query, and the image 2D feature F I is used as the key and value. The image features queried through the deformable attention satisfy the following formula as follows

[0069]

[0070] A mpk = softmax(Linear(f p ))

[0071] Δr mpk = Linear(f p )

[0072] Among them, A mpk and Δr mpk respectively represent the weight and sampling offset of the k-th sampling point in the m-th attention head. W m and W m ' are two learnable weights, M is the number of attention heads, K is the number of sampling points, r p represents the projection point corresponding to the radar point p in the image plane, and Linear() represents a linear mapping layer network.

[0073] Finally, the queried image features are concatenated with the radar point feature f p , and they are fused through an MLP layer to obtain the image-enhanced point feature The formula is as follows:

[0074]

[0075] 4) The grid-guided fusion module performs grid-guided feature fusion based on each 3D proposal box, image 2D features, and radar BEV features to obtain the proposal box features enhanced by the second view features;

[0076] 4) Specifically:

[0077] First, each 3D proposal box is divided into regular grids and the grids are feature-encoded to obtain the grid features of the current 3D proposal box; then the grid features of the current 3D proposal box, as well as the image 2D features and radar BEV features, are used as the input of the first cross-modal deformable attention mechanism, and the first cross-modal deformable attention mechanism is connected to the second cross-modal deformable attention mechanism, and the second cross-modal deformable attention mechanism outputs the proposal box features enhanced by the second view features.

[0078] Specifically as follows:

[0079] The present invention first divides each proposal box into U 3 grids similar to region of interest pooling and generates features in the form of grids, regardless of the presence of radar points. The encoding method of the grid features is as follows:

[0080]

[0081] where δ j = g j - c B is the relative position between the center of each grid and the center c B of the proposed box, g j is the coordinate of the center point of each grid, and c B is the coordinate of the center point of the proposed box, is the number of points in each grid g j , ∈ is a constant offset, and GPE() is the Grid Positional Encoder.

[0082] Next, two cascaded cross-modal deformable attention mechanisms are used to fuse the grid features, image features, and BEV features to obtain the grid features with enhanced two-view features;

[0083] As Figure 3 shown, the initial grid features will be used as a query and fused with the front view (FV) of the image and the BEV features through two cross-modal deformable attention blocks (CMDA) in sequence. The fusion process is similar to the point cloud guidance module. The output of the first CMDA is the front view features of the image queried by each grid g j

[0084] With the front view features of the image the present invention continues to fuse it with the BEV feature F B in the first stage. The front view features of the image are used as the query, and the BEV feature F B is used as the key and value to calculate the fused features through another cross-modal deformable attention (CMDA) block. During this process, the projection point of the grid g j (x j , y j ) on the BEV map is obtained through the following formula:

[0085]

[0086] where X min , Y min are the minimum coordinates on the BEV map, and X size , Y size are the number of grids on the X-axis and Y-axis respectively.

[0087] Through the above steps, the present invention obtains the final features of each grid Then, the U 3 grid features of the proposed box ​They are connected to construct a grid-based proposal box feature where C b is the number of feature channels consistent with

[0088] 5) Feature concatenation and Transformer-based self-attention fusion are performed on the proposal box features enhanced by the two-view features corresponding to each 3D proposal box to obtain the final proposal box features. After the deep learning neural network regresses the offsets based on the final proposal box features and refines and corrects the original proposal boxes based on the obtained offsets, the final 3D object detection boxes are obtained.

[0089] Steps 3)-5) are repeated to process all proposal boxes, and the final 3D object detection boxes corresponding to all proposal boxes are obtained.

[0090] In the first stage of the present invention, radar-guided multi-scale BEV fusion is adopted to provide proposal boxes with high recall rate. In the second stage, based on the point cloud-guided fusion module and the grid-guided fusion module, various forms of features are utilized, including points, front-view images, and BEV, providing comprehensive clues for the refinement of proposal boxes. Experimental results show that the network of the present invention improves the performance of 3D object detection, significantly outperforming the current state-of-the-art methods.

[0091] To evaluate the results of the present invention, experiments are conducted using the publicly available autonomous driving datasets VoD and TJ4DRadSet, and the quantitative results shown in Table 1 and Table 2 are obtained. The evaluation metric is the average precision (Average Precision) of 3D object detection, where the intersection over union (IoU) threshold for cars and trucks is 0.5, and the IoU threshold for pedestrians and cyclists is 0.25. In the experiment, the present invention is compared with two other 4D millimeter-wave radar and image fusion methods. As can be seen from Table 1 and Table 2, regardless of which 3D object detector is used, the 3D object detection performance of the present invention is significantly stronger than the current 4D millimeter-wave radar and image fusion methods. Table 3 shows the ablation experiment results of each modality and each module involved in the present invention. It can be found that the detection performance of the two-modal fusion is better than that of a single one of them. The average metric of the radar-guided BEV multi-scale fusion is improved by 2.6% compared with the direct BEV splicing fusion. After adding the two-stage point cloud-guided fusion module and grid-guided fusion module, the metrics are improved by 2.0% and 3.7% respectively, indicating that the two-stage network plays a role in improving the quality of proposal boxes. In addition, Figure 4 shows the visualization results of the prediction results of several detection examples by the method of the present invention and the LoGoNet method. Figure 4 (a) of Figure 4 is the image, Figure 4The visualization result detected by the present invention is shown in (c). It is a qualitative analysis effect. The figure includes the visualization of the point cloud and the ground truth box in the BEV view, where the blue represents the ground truth box and the red represents the network prediction box. It is not difficult to see that, compared with LoGoNet, the prediction box obtained by the present invention is closer to the ground truth box and can detect more targets. Therefore, it can be concluded that the present invention improves the performance of 3D object detection through the two-stage fusion of 4D millimeter-wave radar and image cross-view.

[0092] Table 1 is a comparison table of 3D object detection performance on the VoD dataset

[0093] Detector Mode Vehicle Index, Pedestrian Index, Bicyclist Index, Average Index SECOND Radar 39.53 36.23 65.68 47.14 PointPillars Radar 41.95 38.09 67.01 49.02 RCFusion Radar + Image 41.70 38.95 68.31 49.65 RCBEVDet Radar + Image 40.63 38.86 70.48 49.99 LXL Radar + Image 42.33 49.48 77.12 56.31 LoGoNet Radar + Image 51.72 49.00 75.54 58.75 The present invention Radar + Image 60.87 57.89 77.46 65.41

[0094] Table 2 is a comparison table of 3D object detection performance on the TJ4DRadSet dataset

[0095]

[0096] Table 3 is a comparison table of the ablation performance of each module of the present invention

[0097]

[0098] The above embodiments are used to explain the present invention rather than limit the present invention. Any modification and change made to the present invention within the spirit and scope of the claims of the present invention fall within the protection scope of the present invention.

Claims

1. A 3D object detection method for 4D millimeter-wave point cloud and image cross-view fusion, characterized in that, It includes the following steps: 1) Use a 3D backbone network to extract voxel features and BEV features from the 4D millimeter-wave radar point cloud, obtaining corresponding multi-scale radar voxel features and multi-scale radar BEV features; and generate the corresponding image 2D features and image BEV features of the image; 2) Use a radar-guided BEV multi-scale fusion module to fuse the radar BEV features and the image BEV features to obtain a BEV fusion feature; then predict 3D proposal boxes based on the BEV fusion feature; 3) The point cloud-guided fusion module performs point cloud-guided feature fusion according to each 3D proposal box, the radar voxel features, and the image 2D features, obtaining the proposal box features after the first view feature enhancement; 4) The grid-guided fusion module performs grid-guided feature fusion according to each 3D proposal box, the image 2D features, and the radar BEV features, obtaining the proposal box features after the second view feature enhancement; 5) Stitch and self-attention fuse the proposal box features after the two-view feature enhancement corresponding to each 3D proposal box to obtain the final proposal box features. After the deep learning neural network regresses the offset based on the final proposal box features and refines and corrects the proposal boxes based on the obtained offsets, the final 3D object detection boxes are obtained.

2. The 3D object detection method for 4D millimeter-wave point cloud and image cross-view fusion according to claim 1, wherein The use of the 3D backbone network to extract voxel features and BEV features from the 4D millimeter-wave radar point cloud, obtaining corresponding multi-scale radar voxel features and radar BEV features, specifically: First, after voxelizing the 4D millimeter-wave radar point cloud, the radar point cloud after voxelization is obtained, and then the SECOND network is used to extract the corresponding multi-scale radar voxel features and BEV features of the radar point cloud after voxelization.

3. A 3D object detection method for 4D millimeter-wave point cloud and image cross-view fusion according to claim 1, characterized in that, The generation of the corresponding image 2D features and image BEV features of the image, specifically: Use a 2D backbone network to extract 2D features of the image, and then perform view transformation on the extracted image 2D features to obtain the image BEV features.

4. A 3D object detection method for 4D millimeter-wave point cloud and image cross-view fusion according to claim 1, characterized in that In the above (2), the use of the radar-guided BEV multi-scale fusion module to fuse the radar BEV features and the image BEV features to obtain the BEV fusion feature, specifically: The BEV fusion feature is obtained by fusing multi-scale BEV fusion features, and the generation steps of each scale of BEV fusion feature are as follows: Predict the BEV occupancy probability map of the current scale according to the radar BEV feature of each scale; then use the occupancy probability map to enhance the image BEV feature to obtain the enhanced image BEV feature; then, fuse the enhanced image BEV feature with the radar BEV feature of the current scale to obtain the BEV fusion feature of the current scale.

5. A 3D object detection method for 4D millimeter-wave point cloud and image cross-view fusion according to claim 1, characterized in that In the above (2), predicting 3D proposal boxes based on the BEV fusion feature, specifically: Input the BEV fusion feature into the proposal box generation network, and the network outputs several 3D proposal boxes.

6. A 3D object detection method for 4D millimeter-wave point cloud and image cross-view fusion according to claim 1, characterized in that, The above (3) is specifically: The point cloud guided fusion module includes a connected cross-modal deformable attention mechanism and region of interest pooling; first, the radar voxel features are fused with the image 2D features through the cross-modal deformable attention mechanism to obtain the point cloud fusion features with enhanced image features Then, the point cloud fusion features with enhanced image features near the current 3D proposal box are aggregated through region of interest pooling The proposal box features with enhanced first view features are obtained.

7. A 3D object detection method for 4D millimeter-wave point cloud and image cross-view fusion according to claim 1, characterized in that The above (4) is specifically: First, divide each 3D proposal box into regular grids and perform feature encoding on the grids to obtain the grid features of the current 3D proposal box; Then, the grid features of the current 3D proposal box, the image 2D features, and the radar BEV features are used as the inputs of the first cross-modal deformable attention mechanism. The first cross-modal deformable attention mechanism is connected to the second cross-modal deformable attention mechanism, and the second cross-modal deformable attention mechanism outputs the proposal box features enhanced by the second view features.

8. A computer device, comprising a memory and a processor, the memory storing a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the 3D object detection method for 4D millimeter-wave point cloud and image cross-view fusion according to any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the 3D object detection method for 4D millimeter-wave point cloud and image cross-view fusion according to any one of claims 1 to 7.

10. A computer program product, comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, it implements the steps of the 3D object detection method for 4D millimeter-wave point cloud and image cross-view fusion according to any one of claims 1 to 7.

Images