Fast image and 4D millimeter wave radar fusion 3D target detection method and device

Abstract

The invention discloses a fast image and 4D millimeter wave radar fusion 3D target detection method and device, and the method comprises the steps: obtaining an RGB image from an image device and a point cloud from a millimeter wave radar, and inputting the RGB image and the point cloud into a trained C4D detection network, thereby enabling the C4D detection network to execute a target detection process, and obtaining a 3D target detection result. The detection process comprises the following steps of: extracting front-view image features from the RGB image, and performing view angle conversion to obtain image BEV features and corresponding BEV heights; extracting radar BEV features from the point cloud; and fusing the image BEV feature and the radar BEV feature to obtain a unified BEV feature, and combining the BEV height to obtain a detection result. According to the invention, when the BEV features of the image are generated, the Height BEVPool is used to retain important height information, the background noise is suppressed, the BEV features of the image and the BEV features of the radar are effectively fused, and the accuracy of 3D target detection is improved.

Description

Technical Field

[0001] This application relates to the field of target detection technology, and more specifically, to a method and apparatus for 3D target detection by fusing fast imagery with 4D millimeter-wave radar. Background Technology

[0002] Environmental perception is a crucial task in autonomous driving. Accurately perceiving environmental information is essential for subsequent decision-making, planning, and control. The perception task encompasses many sub-tasks, including object detection, tracking, and segmentation. 3D object detection, as an important sub-task, aims to perceive the category, location, size, and orientation of each important traffic element in the surrounding environment.

[0003] Cameras, LiDAR, and millimeter-wave radar are widely used in vehicle perception systems, but their different principles of environmental perception lead to numerous differences. LiDAR point clouds contain 3D object structure and accurate depth information, enabling precise environmental perception. However, they are expensive, and their perception capabilities are easily affected by complex weather conditions such as rain, snow, and fog. Millimeter-wave radar has strong penetration, long sensing range, strong environmental adaptability, and low cost. Traditional 3D millimeter-wave radar can only measure distance, horizontal angle, and Doppler velocity, lacking elevation information, which limits its perception performance. 4D millimeter-wave radar adds elevation information compared to 3D millimeter-wave radar, generating 3D point clouds similar to LiDAR, and is gradually being considered a viable alternative. Fusion of cameras and 4D millimeter-wave radar provides an affordable and robust solution. This solution maintains strong adaptability to complex weather conditions and low cost while ensuring perception system performance. Therefore, research on fusion of cameras and 4D millimeter-wave radar for perception is attracting wider attention, and many open-source datasets containing 4D millimeter-wave radar data have been proposed. Since the occlusion problem of objects can be effectively mitigated from the perspective of BEV and is more conducive to the fusion of different modal features, most algorithms convert information from different sensors into BEV features before fusing and detecting them.

[0004] In the process of fusion perception between 4D millimeter-wave radar and cameras, two problems exist regarding height prediction. First, in the acquisition of image BEV (Body Estimation Value), the widely used LSS-based and sampling-based view transformation methods often involve BEV pooling operations that accumulate features along the height dimension. This operation significantly loses height information, making it difficult to predict height from the accumulated features. Second, although 4D millimeter-wave radar possesses elevation information, the height information of the point cloud is not accurate enough due to the influence of the aperture, exacerbating the inaccuracy of height prediction. Summary of the Invention

[0005] This application provides a fast image fusion and 4D millimeter-wave radar method and apparatus for 3D target detection, to improve the accuracy of 3D target prediction. The specific scheme is as follows:

[0006] In a first aspect, this application provides a fast 3D target detection method that fuses images with 4D millimeter-wave radar, including:

[0007] Acquire RGB images from imaging devices, point clouds from millimeter-wave radar, and a trained C4D detection network;

[0008] Both the RGB image and the point cloud are input into the trained C4D detection network to perform an object detection process and obtain the detection result of the 3D object. The object detection process is as follows:

[0009] Forward view image features are extracted from the RGB image; the forward view image features are transformed to obtain image BEV features and corresponding BEV height; radar BEV features are extracted from the point cloud; the image BEV features and the radar BEV features are fused into unified BEV features; 3D target detection is performed by combining the BEV height corresponding to the image BEV features and the unified BEV features to obtain the detection result of the 3D target.

[0010] Secondly, this application provides a rapid image and 4D millimeter-wave radar fusion 3D target detection device, characterized in that it includes an acquisition module and a detection module;

[0011] The acquisition module is configured to acquire RGB images from an image device, point clouds from a millimeter-wave radar, and a trained C4D detection network.

[0012] The detection module is configured to input both the RGB image and the point cloud into the trained C4D detection network, so that it performs a target detection process to obtain the detection result of the 3D target. The target detection process is as follows:

[0013] Forward view image features are extracted from the RGB image; the forward view image features are transformed to obtain image BEV features and corresponding BEV height; radar BEV features are extracted from the point cloud; the image BEV features and the radar BEV features are fused into unified BEV features; 3D target detection is performed by combining the BEV height corresponding to the image BEV features and the unified BEV features to obtain the detection result of the 3D target.

[0014] The innovative aspects of this application's embodiments include:

[0015] 1) This application proposes a Height BEVPool (BEV pooling with height prediction) scheme to address the issues of loss of crucial height information during image BEV acquisition and inaccurate bounding box height prediction due to insufficient height accuracy of 4D millimeter-wave radar. This effectively improves perception performance while only adding a negligible amount of inference latency. It can be easily extended to any existing LSS-based or sampling-based methods.

[0016] 2) This application proposes a plug-and-play PVHG module that requires no additional labels other than the 3D truth box, and achieves the function of suppressing BEV background noise generation and improving the height prediction accuracy of Height BEVPool. It is simple but effective.

[0017] 3) This application proposes an IGF module to improve the robustness of fusion and suppress BEV background noise. HeightBEVPool is applied to the sampling strategy to obtain image BEV features. A lightweight yet high-performance model is established by combining PVHG and IGF to improve the accuracy of predicting 3D targets. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0019] Figure 1 A flowchart illustrating a fast image-to-4D millimeter-wave radar fusion 3D target detection method provided in this application embodiment;

[0020] Figure 2 An architecture diagram of the C4D detection network provided in the embodiments of this application;

[0021] Figure 3 This is a structural diagram of the PVHG module provided in an embodiment of this application;

[0022] Figure 4a The original image input to the PVHG module provided in the embodiments of this application;

[0023] Figure 4b The comparison diagram of PVHG before and after booting is provided for the embodiments of this application;

[0024] Figure 5 A structural diagram of the high probability sampling module provided in the embodiments of this application;

[0025] Figure 6A truth box height distribution diagram provided for embodiments of this application;

[0026] Figure 7 A schematic diagram of the IGF module provided in an embodiment of this application;

[0027] Figure 8 This is a schematic diagram of the structure of the detection head provided in the embodiments of this application;

[0028] Figure 9a A schematic diagram of the projection of the truth box of the VoD dataset provided in this embodiment onto an image;

[0029] Figure 9b This is a visualization diagram of the detection results of the VoD dataset provided in the embodiments of this application;

[0030] Figure 10a This application provides a schematic diagram of the projection of the truth boxes of the TJ4DRadset dataset onto an image for the purposes of this embodiment.

[0031] Figure 10b A visualization diagram of the detection results of the TJ4DRadset dataset provided in the embodiments of this application. Detailed Implementation

[0032] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0033] It should be noted that the terms "comprising" and "having," and any variations thereof, in the embodiments and accompanying drawings of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the steps or units listed, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices.

[0034] To address the shortcomings of existing technologies, this application proposes a method that, while pooling features along the height dimension, predicts the height of each BEV grid to maximize the retention of height information from the established 3D feature point cloud. This predicted height information is then used to assist in the final bounding box height prediction. For LSS-based algorithms, the 3D feature point cloud represents view frustum features; for sampling-based methods, it represents a series of pre-defined sampling points. Furthermore, this application proposes a PVHG (Perceptive View Heatmap Guidance) module, which guides the foreground image feature map to focus more on foreground objects, suppressing BEV background noise. It also makes the distribution of the foreground feature map more concentrated in the height direction, assisting BEV pooling for more accurate height prediction. In addition, this application proposes an Interactive Guidance Module (IGF) to improve the robustness of multimodal fusion perception and suppress BEV background noise. The method described in this application can be easily applied to pure vision or the fusion of vision and point clouds.

[0035] This application discloses a fast image-to-4D millimeter-wave radar fusion method and apparatus for 3D target detection. It acquires an RGB image from an image device, a point cloud from a millimeter-wave radar, and a trained C4D detection network. Both the RGB image and the point cloud are input into the trained C4D detection network to perform a target detection process and obtain the 3D target detection result. The method involves extracting front-view image features from the RGB image; performing a viewpoint transformation on the front-view image features to obtain image BEV features and the corresponding BEV height; extracting radar BEV features from the point cloud; fusing the image BEV features and radar BEV features into a unified BEV feature; and combining the BEV height corresponding to the image BEV features and the unified BEV feature to perform 3D target detection, thus obtaining the 3D target detection result. This application uses a Height BEVPool to retain important height information and suppress background noise during image BEV feature generation, and effectively fuses image BEV features and radar BEV features, improving the accuracy of 3D target detection.

[0036] The embodiments of this application will be described in detail below.

[0037] Figure 1 This document illustrates a flowchart of a fast image-to-4D millimeter-wave radar fusion 3D target detection method provided in an embodiment of this application. Figure 1 As shown, the purpose of this application is to predict a bounding box containing the 3D target for each 3D target.

[0038] This application provides a fast image-to-4D millimeter-wave radar fusion 3D target detection method, which includes:

[0039] S1, acquire RGB images from an image device, point clouds from a millimeter-wave radar, and a trained C4D detection network;

[0040] The process of training the C4D detection network in this application is the same as the training process of training neural network models in the prior art, and will not be described in detail here.

[0041] S2, input both the RGB image and the point cloud into the trained C4D detection network to perform the object detection process and obtain the detection results of 3D objects. The object detection process is as follows:

[0042] Forward view image features are extracted from RGB images; the forward view image features are transformed to obtain image BEV features and corresponding BEV heights; radar BEV features are extracted from point clouds; image BEV features and radar BEV features are fused into unified BEV features; 3D target detection is performed by combining the BEV heights corresponding to the image BEV features and the unified BEV features to obtain the 3D target detection results.

[0043] In one specific embodiment of this application, reference is made to Figure 2 As shown, the trained C4D detection network consists of an image encoder, a viewpoint transformation module, a radar encoder, an interactive guidance module, and a detection head connected in sequence.

[0044] An image encoder is used to extract front view image features from an input RGB image;

[0045] exist Figure 2 In this model, the image encoder consists of a backbone network and a neck network connected in sequence. The backbone network extracts multi-scale forward-looking features from the input RGB image, and the neck network fuses the multi-scale forward-looking features to obtain forward-looking image features.

[0046] Define RGB image as Where N represents the number of cameras, 3 represents the RGB channels of the image, and H i ×W i Given the input image size, the backbone network first extracts multi-scale forward-looking features, and then the neck network fuses these features from different scales to obtain the forward-looking image features. C P This represents the number of channels in the front-look feature map. The backbone uses v2-99, i.e., VoVNet2, where 99 is the structural parameter. A pre-trained model specifically trained for depth estimation tasks will be used to initialize the backbone network and freeze the weights during training. The (Neck) uses Freq FPN, which differs from FPN in that FreqFusion replaces the upsampling process in the original FPN to better fuse multi-scale features and increase the accuracy of subsequent depth estimation.

[0047] The perspective transformation module is used to transform the front view image features to obtain the image BEV features and the corresponding BEV height;

[0048] A radar encoder is used to extract radar BEV features from an input point cloud.

[0049] The interactive guidance module is used to fuse image BEV features and radar BEV features to obtain a unified BEV feature through interactive guidance.

[0050] The detection head is used to perform 3D target detection by utilizing the BEV height corresponding to the BEV features of the image and the uniform BEV features, and obtain the detection results of the 3D target.

[0051] In one specific embodiment of this application, the viewpoint transformation module includes a depth network, a PVHG module, and a height probability sampling module;

[0052] The deep network predicts the discrete depth distribution of each feature pixel based on the features of the front view image, and outputs the discrete depth distribution and the front view image features with the number of channels compressed.

[0053] DeepNet will be based on I P To predict the discrete depth distribution of each feature pixel, denoted as... Where D N This represents the number of discrete depths. Additionally, DepthNet outputs the front view image features with compressed channel counts, denoted as... Where C DP This represents the number of channels for the front view image features output by DepthNet. The DepthNet in this application is essentially consistent with the BEVDepth method (a prior art method), also using a LiDAR depth map as supervision; the difference lies in the number of input and output channels. PVHG will be used for I... DP Adjustments are made to make it focus more on foreground objects, so that the subsequently generated BEV features are more focused on foreground objects. This also makes the feature value distribution in the height direction more concentrated, which helps improve the accuracy of the height predicted by the subsequent Height BEVPool. The detailed structure of PVHG will be introduced in III-B. The foreground features adjusted by PVHG are denoted as... Where C RP This represents the number of channels in the front-look feature map output by the PVHG. Height ProbSampling combines the discrete depth distribution D with camera intrinsic and extrinsic parameters to remove I. RP The features of preset points in the BEV space are obtained by mid-sampling, and finally, the core operation proposed in this application—Height BEVPool—will simultaneously obtain the image BEV features and BEV height.

[0054] Most existing camera-based 3D object detection technologies convert foreground image features into BEV (Browser Object Detection) features for detection. Based on the BEV acquisition method, these can be categorized into LSS (Laser Subtraction Slice) based methods, sampling-based methods, and sampling plus LSS methods. LSS first estimates the discrete depth distribution of each pixel in the foreground feature map, then lifts the foreground feature map to 3D space, and finally accumulates features from the same volumetric cylinder to obtain the final BEV features. Both BEVDepth and BEVStereo utilize LiDAR data as ground truth depth-supervised deep networks to improve performance. BEV-SAN proposes a Slice Attention Module to fuse BEV slices of different heights, corresponding to the heights of different categories. SA-BEV proposes SA-BEVPool to suppress the generation of background BEVs through foreground object probability thresholds and depth score thresholds. Sampling-based methods first pre-define a series of points in 3D space, then aggregate features near the corresponding projection points on the foreground feature map to obtain features from 3D sampling points, and finally compress them into BEV features. Besides generating BEV features, HeightFormer also predicts the height of each BEV grid, effectively improving model performance. However, its deformable attention mechanism, which relies on viewpoint transitions, has high computational overhead. Both FB-BEV and DualBEV employ LSS upsampling to compensate for the sparse feature points at distant locations in the LSS method. The difference lies in the fact that, unlike the deformable attention sampling mechanism used in FB-BEV, DualBEV directly uses the forward-looking features corresponding to the projection points as features for the 3D sampling points, allowing the sampling process to be accelerated using BEVPoolv2. Furthermore, both DualBEV and FB-BEV use BEVMask ground truth values ​​as supervision to suppress background noise in the BEV. These ground truth BEVMask values ​​are obtained by projecting the ground truth boxes onto the BEV viewpoint.

[0055] The aforementioned existing methods have two problems. First, both sampling and LSS methods require using BEVPool or BEVPoolv2 to "splat" 3D spatial features into BEV features. This operation significantly loses the height information of the 3D spatial features. To better utilize 3D spatial features, this application proposes Height BEVPool based on BEVPoolv2. While accumulating features within the same BEV grid, it predicts the height corresponding to each BEV grid, greatly improving model performance while maintaining high speed. Second, SA-BEVPool and DualBEV require using the network's predicted foreground object probability as the criterion for determining whether an object is foreground when suppressing background noise in the foreground feature map. The corresponding network is supervised by LiDAR point cloud category labels. However, LiDAR point cloud category labeling is labor-intensive and costly.

[0056] To address this issue, this application proposes the PVHG (PV Heatmap Guidance module), which uses only the front-looking 2D ground truth bounding box to guide the front-looking feature map to focus more on foreground objects and suppress background features. Simultaneously, PVHG also makes the feature values ​​more concentrated in the height direction, which is beneficial for Height BEVPool to perform more accurate height prediction. Both PVHG and Height BEVPool can be easily integrated into existing LSS-based or sampling-based methods.

[0057] The PVHG module uses the number of channels in the front view image features after the number of channels is compressed to predict the front view image heatmap for each category; guided by the front view image heatmap, it combines the front view features after the number of channels is compressed to output the adjusted front view image features.

[0058] refer to Figure 3 As shown, accurately regressing the 2D center of the foreground object from the BEV feature requires the generated BEV feature to focus more on the foreground object and suppress background noise. Many other existing methods directly supervise the final output of the detector head, which is not effective enough. Both DualBEV and SA-BEV predict the probability that each foreground feature pixel is a foreground object, and then use the predicted probability to guide the foreground feature map to focus more on the foreground object. Their experimental results demonstrate that guiding the foreground feature to focus more on the foreground object before converting the foreground feature to BEV feature does indeed help improve model performance. However, their approach requires supervision using ground truth labels for point cloud categories, which is costly and cumbersome.

[0059] refer to Figure 3 As shown, Figure 3The structure of the PVHG module. Input front view image features I DP First, the number of channels is compressed to a certain dimension and the receptive field is increased by sequentially passing the data through 3×3 convolutional layers and ReLU activation function. Then, 1×1 convolutions are used to predict the front view image heatmap PVH for each category.

[0060] PVH=C1(R(C3(I DP )))

[0061] Where C1 and C3 represent 1×1 and 3×3 convolutional layers, respectively, and R represents the ReLU activation function. Gaussian Focal Loss is applied to the lookahead heatmap for each category for supervision. PVH and I DP Performing an outer product yields front-look features guided by different categories. These front-look features from different categories are then concatenated along the channel dimension and compressed to C using a 3×3 convolution. DP , and then with I DP The summation yields the front view image feature I guided by the front view image heatmap of all categories. RP That is, the adjusted front view image features:

[0062]

[0063] The brackets "[]" indicate channel splicing operations. The heatmap for each category and I DP Perform the outer product separately.

[0064] Refer to Figure 4 and Figure 4b , Figure 4a The original image is displayed in the image. Figure 4b The image front-look feature maps before and after using PVHG are shown. Figure 4b The left image shows the front-view feature map before PVHG guidance, and the right image shows the front-view feature map after PVHG guidance. It can be observed that the feature map after PVHG guidance focuses more on foreground objects, which helps the network distinguish foreground objects from the background. Simultaneously, the feature values ​​for height are more concentrated, improving the accuracy of the height predicted by the subsequent Height BEVPool.

[0065] The height probability sampling module, combining discrete depth distribution and the intrinsic and extrinsic parameters of the image device, samples the features of preset points in the BEV space from the adjusted front view image features to obtain the image BEV features and the corresponding BEV height.

[0066] Existing common viewpoint transformation methods include LSS-based and sample-based methods, both of which can obtain unified BEV features from the image's forward-looking features for object detection. However, both methods compress the height dimension information, resulting in BEV features that primarily represent information such as object location, orientation, and size. Accurately regressing object height from such BEV features becomes difficult. To address this issue, this application proposes Height BEVPool based on BEVPoolv2. In addition to acquiring BEV features with compressed height dimensions, it also predicts BEV Height, representing the height corresponding to the presence of an object in each predicted BEV grid. This prediction is used to assist the final detection head in regressing and predicting object height. Height BEVPool can be easily embedded into existing LSS or sample-based algorithms. However, since previous work, such as LXL, has demonstrated the superior performance of sample-based methods (which will be confirmed by subsequent ablation experiments), this application chooses the sample-based method to obtain BEV features and BEV height. The feature sampling branch of DualBEV replaces the grid sampling method with the round() method to establish a one-to-one correspondence between sampling points and the indices of the front-look feature map and the depth map, allowing the sampling process to be accelerated using BEVPoolv2. Based on this, this application introduces Height BEVPool, named Height Prob Sampling. The operation of Height Prob Sampling is as follows: Figure 5 As shown.

[0067] In one specific embodiment of this application, the high probability sampling module is used for:

[0068] a. Set sampling points;

[0069] Some three-dimensional spatial points will be pre-set, denoted as S. R ∈{(x j ,y j ,z j Let |j=1,2,…,n}, where n=N×N Z ×H B ×W B Represents the total number of preset points, with the superscript R indicating the radar coordinate system space, and H... B With W B N represents the height and width of the BEV feature map, respectively. ZThe number of sampling points in the height direction is represented by N, which represents the number of cameras. Since the pixel values ​​of the forward-looking features at different heights vary significantly, a single, sparse 3D sampling point setup will easily miss sampling features of small objects and fail to distinguish the importance of features in different height ranges. DualBEV employs a multi-resolution sampling point strategy to address this issue. It sets sampling points at 0.5m intervals within the ROI region [-2,2], and then superimposes these points with sampling points set at 1m intervals throughout the entire height range [-5,3]. This application adopts this multi-resolution sampling strategy, maintaining the same number of sampling points. However, unlike the TJ4Dradset dataset, the VoD dataset and TJ4Dradset dataset processed in this application have entire height ranges of [-3,2] and [-4,2], respectively, instead of [-5,3] on the nuscenes dataset. Therefore, the sampling interval in this application will be smaller. It is worth noting that the height coordinates of these sampling points will subsequently be used as the target height prediction value for each BEV grid when an object exists, which is the BEV Height mentioned above. Therefore, this application adopts a denser sampling point setting strategy to improve the height resolution of the sampling points, thereby improving the accuracy of height prediction.

[0070] Although the total number of sampling points is determined, the extent of dense sampling is a problem that needs to be considered. This application statistically analyzes the center distribution of the bounding boxes in the VoD dataset and the TJ4Dradset dataset, respectively, and the results are as follows: Figure 6 As shown. The height of the sampling points should be set to match the height of the object's center as closely as possible; therefore, in conjunction with... Figure 6 The statistical results determined the final sampling point settings as follows: For the VoD dataset, 5 sampling points were set at intervals of 0.625m within the ROI height range [-1.125, 1.375], and superimposed on 8 sampling points set at intervals of 0.625 within the entire height range [-2.6875, 1.6875]; For the TJ4Dradset dataset, 5 sampling points were set at intervals of 0.75m within the ROI height range [-3.25, -0.25], and superimposed on 8 sampling points set at intervals of 0.75 within the entire height range [-3.625, 1.625].

[0071] b. Using the intrinsic and extrinsic parameters of the image device, establish the correspondence between the sampling points and the depth map and image feature map, respectively;

[0072] Each point in S First, the camera's intrinsic parameter matrix K and extrinsic parameter matrix T are projected onto the image coordinate system. Then, points outside the view frustum are filtered out to obtain the effective sampling point set S. I ∈{(u i ,v i ,1)·di |i=1,2,…,n v}:

[0073]

[0074] Where the superscript I represents the image coordinate system space, u i and v i d represents non-integer pixel coordinates. i n represents a non-integer depth value. v Γ represents the number of sampling points falling within the view frustum, and Γ represents the operation of filtering out points not within the view frustum. H is obtained by retaining the z-axis coordinate of each valid point in the radar coordinate system. R ∈{(z i | i = 1, 2, ..., n v}:

[0075]

[0076] Here, `remain_z` indicates that only the z-axis coordinate in the radar coordinate system is retained. It also retains the BEV grid index B corresponding to the valid sampling points. ind Then, for the coordinates of each valid point... Performing the rounding operation, i.e., the get integer operation in Figure 4, yields the integer coordinate values ​​in the depth map space.

[0077]

[0078] Here, `round()` represents the `torch.round()` method, which uses rounding to accelerate feature sampling during inference using `BEVPoolv2`. `clip` represents the operation of limiting the value range. `d1` represents the depth value of the nearest section of the view frustum, `d2` represents the depth value of the farthest section of the view frustum, `Δd` represents the unit length of the discrete depth distribution in the depth direction, and `H`... i and W i This represents the size of the input image. Then, similar to BEVPoolv2, it is first determined based on... The depth map index of each sampling point is calculated. With feature map index Establish a one-to-one correspondence between sampling points and feature maps F and depth maps D, based on Sort by the size of the value. and Obtain the Index Table.

[0079] c. Using the discrete depth distribution and the adjusted front view image features, calculate the feature value of each valid sampling point;

[0080] For ease of understanding, Figure 5 The Height BEVPool only demonstrates the calculation process for a single BEV feature channel and BEV Height channel. The final number of channels for BEV features and BEV Height will be related to I. RP Maintain consistency. Input one channel I of the front view image feature map. RP-k With depth map D, the feature value of each valid sampling point To be calculated:

[0081]

[0082] in and These represent the i-th sampling point in the valid sampling point sequence, processed according to the corresponding depth map index and feature map index in the Index Table, respectively, for D and I. RP-k The depth score and feature values ​​of the front view feature map obtained in the process.

[0083] d. Accumulate the feature values ​​of the effective sampling points in each BEV grid to obtain the feature of each BEV grid, and combine the features of all BEV grids to form the BEV feature of the image.

[0084] The BEV feature of the k-th channel image is obtained by accumulating the feature values ​​of the effective sampling points in each BEV grid.

[0085]

[0086] Where j represents the BEV grid number, L s The variable represents the number of valid sampling points contained in the j-th BEV grid, p represents the valid sampling point number, and sum represents the summation operation.

[0087] e. For the same channel containing all valid sampling points in each BEV grid, select the height of the valid sampling point corresponding to the largest feature value in the radar coordinate system and determine it as the BEV height in that channel; combine the BEV heights of all channels in each BEV grid to form the height of the BEV grid; combine the heights of all BEV grids to form the BEV height corresponding to the BEV feature of the image.

[0088] The predicted BEV Height for the j-th BEV grid is the height of the sampling point corresponding to the largest eigenvalue within the grid in the radar coordinate system.

[0089]

[0090] Ultimately, the BEV features of the image will be obtained. With BEV Height BEV Height represents the height of an object when it is present in each predicted BEV grid, and it will be used to assist the detection head in regressing the height of the predicted bounding box. Since the added BEV Height prediction operations are mostly calculated in the CUDA code, the model will be able to retain the crucial height information that is missing from the BEV perspective with very little computational cost.

[0091] In one specific embodiment of this application, a radar encoder is used to perform volumetric columnarization on the input point cloud, and then encode each non-empty volumetric column to obtain pseudo-image features of the point cloud. Multi-scale features are extracted from the pseudo-image features, and the multi-scale features are fused to obtain the BEV features of the 4D millimeter-wave radar, which are then used as the radar BEV features.

[0092] On the one hand, the radar BEV feature F obtained from the Radar Encoder BEV-R Image BEV features F obtained from View Transformer BEV-C Since different sensor modes will have differences, directly stitching together features along the channel dimension for fusion will yield poor results. Furthermore, background noise in the BEV mesh will affect detection performance. To address these issues, inspired by DualBEV and RCFusion, this application proposes an Interactive Guidance Fusion (IGF) module to fuse features from different modes.

[0093] Since 4D millimeter-wave radar datasets have only been released in recent years, there are relatively few algorithms for 3D target detection by fusing images with 4D millimeter-wave radar data. RCFusion proposes a shared attention encoder to fuse multiple BEV feature maps sampled from different front-view image resolutions into a single image BEV feature. It also proposes an attention-based IAM module to suppress the data modality discrepancy problem that exists when directly using convolutional layers to fuse BEV and radar features. LXL proposes a "radar occupancy-assisted depth-based sampling" method to sample multi-scale front-view feature maps, effectively using radar information for image feature sampling. However, its direct fusing of image BEV and radar BEV through convolution ignores the problem of data modality inconsistency. UniBEVFusion incorporates depth information from the millimeter-wave radar point cloud projected onto the image, as well as additional channel information (such as velocity, RCS, etc.), before performing pixel-by-pixel depth estimation to improve the robustness and accuracy of depth estimation, while providing richer information to the front-view image feature map. Similar to RCFusion's IAM, UniBEVFusion also proposes a UFF module to mitigate differences between different modalities and improve the robustness of multimodal fusion.

[0094] Both RCFusion's IAM and UniBEVFusion's UFF require multiplying the different modal BEV features input by a weight layer to enhance foreground information from the BEV perspective, but lack foreground BEV ground truth supervision. Inspired by DualBEV, this paper proposes an IGF (interactive guidance fusion) module, which guides image BEV features through BEV foreground probability weights jointly generated by the image and radar, thereby enhancing the accuracy and robustness of image BEV feature fusion. Simultaneously, a BEV Mask obtained by projecting the 3D ground truth box onto the BEV perspective is used for supervision. Then, the image BEV features guided by the BEVMask are used to guide the millimeter-wave radar BEV, thereby suppressing interference noise from the millimeter-wave radar.

[0095] In one specific embodiment of this application, reference is made to Figure 7 As shown, the interactive guidance module includes: RAG module, IGM module, and splicing module;

[0096] Among them, the RAG module predicts the occupancy probability of the BEV network through image BEV features and radar BEV features, and uses the occupancy probability to guide the image BEV features, resulting in image BEV features that have been jointly guided and adjusted by image BEV features and radar BEV features.

[0097] The RAG module will first pass through F BEV-R With F BEV-C To predict the BEV grid occupancy probability P BEV And used to guide F BEV-C Obtained through F BEV-C With F BEV-R Image BEV features jointly guided and adjusted

[0098]

[0099] CBR i This represents a 3×3 convolution, with the BatchNorm layer and ReLU activation function component used to adjust the number of channels. This indicates that SAE-ProbNet is used to predict the BEV mesh occupancy probability, and will use the foreground object BEV mask for supervision.

[0100] The IGM module uses image BEV features to guide radar BEV features, thus obtaining radar BEV features guided by image BEV features.

[0101] The Image Guidance Module (IGM) will then use Used to guide FBEV-R Obtain radar BEV features guided by image BEV features.

[0102]

[0103] Where SE stands for SELayer, which is used to adjust the importance of different channels of image features, MP and AP represent channel max pooling and channel average pooling, respectively, C3 represents 3×3 convolution, and σ represents sigmoid activation function.

[0104] The stitching module stitches and convolves the guided image BEV features with the guided radar BEV features to obtain unified BEV features.

[0105] The stitching module will use the guided image BEV features Radar BEV characteristics The final BEV features F are obtained by concatenating and convolution. BEV :

[0106]

[0107] The RAG module in this application considers BEV features of different modalities to predict the BEV grid occupancy probability, which effectively suppresses background noise in the image BEV features. Furthermore, the image BEV features after background noise suppression... The guidance can alleviate the problem of noise point cloud interference in 4D millimeter-wave radar.

[0108] In one specific embodiment of this application, the detection head is used to perform convolution, normalization, and nonlinear processing on uniform BEV features to obtain intermediate BEV features; adjust the number of channels of the BEV height corresponding to the image BEV features to obtain channel-adjusted BEV height features; concatenate the channel-adjusted BEV height with the intermediate BEV features along the channel dimension, and adjust the channel dimension again after concatenation to obtain the final height feature; using the final height feature and the intermediate BEV features, multi-dimensional information of the prediction box labeled with the 3D target location is predicted. The multi-dimensional information includes geometric center coordinates, length, width, height, and orientation angle.

[0109] The detection head Height CenterHead in this application is obtained by making minor modifications to the CenterPoint detection head, and its network structure is as follows: Figure 8 As shown, the blue section represents the changes. The BEV Height, after channel number adjustment, will first be synchronized with CBR. 4 The output features are concatenated along the channel dimension, then the channel dimension is adjusted again, and finally fed into the height header to regress the final predicted height of each BEV grid:

[0110]

[0111] Where C1 represents a 1×1 convolution, used to adjust the number of feature channels to 32, which is CBR. 4 Half of the output, This indicates the original CenterHead height prediction header. The prediction of other bounding box information remains consistent with the CenterHead. Height CenterHead will effectively improve the accuracy of bounding box height prediction.

[0112] This application provides a fast image and 4D millimeter-wave radar fusion 3D target detection device, which includes an acquisition module and a detection module;

[0113] The acquisition module is configured to acquire RGB images from an image device, point clouds from a millimeter-wave radar, and a trained C4D detection network.

[0114] The detection module is configured to input both RGB images and point clouds into the trained C4D detection network, enabling it to perform an object detection process to obtain the detection results of 3D objects. The object detection process is as follows:

[0115] Forward view image features are extracted from RGB images; the forward view image features are transformed to obtain image BEV features and corresponding BEV heights; radar BEV features are extracted from point clouds; image BEV features and radar BEV features are fused into unified BEV features; 3D target detection is performed by combining the BEV heights corresponding to the image BEV features and the unified BEV features to obtain the 3D target detection results.

[0116] To demonstrate the effectiveness of the proposed solution, a dataset is used to verify its validity.

[0117] The VoD and TJ4DRadSet datasets will be used to validate the effectiveness of the proposed algorithm. Both are designed for autonomous driving and have a data format similar to the KITTI dataset, but unlike KITTI, they both include 4D millimeter-wave radar information. Each labeled object in the scene includes corresponding category information, 3D bounding box, Track ID, etc. For the VoD dataset, following the official dataset partitioning method, the training set and validation set will contain 5139 frames and 1296 frames of data, respectively. Five frames of cumulative point cloud and image data will be used for training and validation. For the TJ4DRadSet dataset, following the official partitioning method, the training set and validation set will contain 5717 frames and 2040 frames of data, respectively.

[0118] AP will be used to evaluate the results, and mAP will be calculated by averaging the APs of multiple categories. For the VoD dataset, the same evaluation method as the official dataset will be used, and the evaluation is divided into the ROI region and the entire annotated region. The evaluation categories are Car, Pedestrian, and Cyclist, and the corresponding IoU thresholds are 0.5, 0.25, and 0.25 respectively. The ROI region represents the region {(x, y, z)| -4 < x < 4m, z < 25m} in the camera coordinate system. For the TJ4DRadSet, objects within 70m in front in the radar coordinate system will be evaluated. In addition to Car, Pedestrian, and Cyclist, the evaluation category also includes Truck, and its IoU threshold is 0.5.

[0119] For the VoD dataset, the input image resolution of the image branch is 320×960. After passing through v2-99 and FreqFPN, the resolution of the obtained feature map is defaulted to 1 / 8 of the original. The resolutions of the image BEV feature map and BEVHeight obtained through VT are both 160×160, and the number of channels is both 80. The 4D millimeter-wave radar point cloud range is restricted within the radar coordinate system {(x, y, z)| 0 < x < 51.2m, -25.6 < y < 25.6m, -3 < z < 2m}, and the {x, y, z, RCS, v, v c , t} channels of the point cloud will be used, where x, y, z are the point cloud coordinates, RCS is the reflection intensity, v is the relative Doppler velocity, v c is the corrected Doppler velocity, and t is the timestamp. The size of the volume column is 0.16m×0.16m, and the resolution of the radar BEV feature output by SECONDFPN is also 160×160, and the number of channels is 384. The number of channels of the unified BEV feature finally output by IGF is 256.

[0120] For the TJ (TJ) 4DRadSet dataset, its input resolution is 960×1280. The resolutions of the image BEV feature map and BEV Height obtained through VT are both 216×248. The millimeter-wave radar point cloud range is {(x, y, z)| 0 < x < 69.12m, -39.68 < y < 39.68m, -4 < z < 2m}, and the {x, y, z, power, v, v c} channels of the point cloud will be used, where power represents the signal-to-noise ratio. It should be noted that there is no v c in the millimeter-wave radar point cloud feature channels of TJ4DRadset. This application follows the method of Bin Tan et al. in the prior art to obtain v c . Other network parameter configurations are the same as those implemented on VoD.

[0121] This application utilizes the MMDetection3D framework. Model training was performed on two NVIDIA GeForce RTX 4090D graphics processing units (GPUs), and accuracy evaluation was performed on one 4090D GPU. Model inference speed evaluation was performed on one RTX 3090 GPU. Model training was performed end-to-end, using AdamW as the optimizer, with a total of 20 epochs, a batch size of 6, and a learning rate training strategy of starting from 2e for the first 200 batches. -6 Linear growth to 2e -4 Then, the learning rate was decayed to 2e using CosineAnnealingLR. -8 During training, image data is augmented using random scaling and random flipping, while point cloud data is augmented using random scaling and random horizontal flipping. Point clouds and ground true bounding boxes not falling within the view frustum are filtered out. In PVHG, the minimum overlap of GaussianFocalLoss is set to 0.0001 to result in a larger calculated Gaussian radius, making the guidance more gentle and avoiding suppression of important pixel features around the front-view feature pixels where the object center falls, thus affecting performance. The minimum Gaussian radius used to generate the ground truth heatmap in HeightCenterHead is 2. Both image and point cloud data are normalized before being fed into the network.

[0122]

[0123] Where i m p represents the pixel value of each channel in an RGB image. n This represents the different channel values ​​for each point cloud. and These represent the mean values ​​of the image and point cloud for different channels across the entire dataset, respectively. and These represent the standard deviations of the image and point cloud across different channels in the entire dataset.

[0124] During inference, for the VoD dataset, images are first scaled to 0.48 of their original size and then cropped or padded to 320×960 before being fed into the network. For the TJ4DradSet dataset, images are directly fed into the network at a size of 960×1280. The top 1000 detection results from both datasets are considered, followed by NMS for post-processing. The minimum radii for Car, Pedestrian, Cyclist, and Truck are 4m, 0.3m, 0.85m, and 12m, respectively.

[0125] Table I shows the algorithm performance on the VoD dataset.

[0126]

[0127]

[0128] Table 1 shows the comparison between the proposed method and other algorithms on the VoD validation set, with gray representing the experimental results of the proposed method. C4D-R is the point cloud branch of this proposed method, and its structure is similar to that of LXL-R, thus its performance is similar. C4D-8x is the default network configuration of this proposed method, indicating that the image BEV features are sampled from the front-look feature map at an 8x downsampling, while C4D-4x indicates that the image BEV features are sampled from the image front-look feature map at a 4x downsampling. Comparing C4D-R and C4D-8x, it can be seen that after fusing image information, the model performance improved by 9.95% and 7.00% in the entire labeled area (EAA) and driving area (DC), respectively. Small objects showed significant improvements, with pedestrians and cyclists in the driving area increasing by 10.64% and 7.33%, respectively, and pedestrians and cyclists in the entire labeled area increasing by 12.08% and 5.29%, respectively. This indicates that dense semantic features of the image help compensate for the sparse point cloud of 4D millimeter-wave radar, especially improving the detection capability of small objects. The smaller increase in the detection of cyclists compared to pedestrians demonstrates that pure 4D millimeter-wave radar already possesses good moving object capture capabilities. RCFusion and LXL sampled on front-look feature maps at different scales, while the model in this application (even the relatively lightweight C4D-8X) only needs to sample on a single front-look feature map to achieve better mAP in both the entire labeled area and driving area, proving the effectiveness of the proposed method. Similar to BEVFusion, UniBEVFusion expands the point cloud features into specific channels after projecting them onto the forward-looking feature map, then stitches them together with the forward-looking feature map before performing depth estimation. However, due to the increased noise, sparser point cloud, and inaccurate height of radar point clouds, this method of directly copying the fusion of LiDAR and camera does not perform ideally. Furthermore, although it uses the more feature extraction-capable Swing Transformer as the image feature encoder, its mAP in both the entire labeled area and the driving area is still lower than that of C4D-8x.

[0129] This application also specifically evaluated the inference speed of the model on the Nvidia RTX 3090 GPU, and the results are shown in the last column of Table 1. The pure point cloud modality can run at a very high speed. The inference speed drops significantly after adding dense image data, but both the C4D-8x's 21.4fps and the C4D-4x's 18.2fps still have extremely high real-time performance compared to the LXL's 6.1fps.

[0130] Figure 9a and Figure 9b The detection results of C4D and C4D-R on the VoD dataset were visualized. Figure 9a It is the projection of the truth box onto the image. Figure 9b The first row shows the detection results of C4D-8x, and the second row shows the detection results of C4D-R. As can be seen, C4D can effectively avoid false detections by using image information to suppress point cloud noise from 4D millimeter-wave radar. Furthermore, C4D can utilize the excellent velocity capture capabilities of 4D millimeter-wave radar for more accurate detection, such as the cyclist located further to the right of the image in the second column, occupying very few pixels.

[0131] Table 2 TJ4Dradset Performance

[0132]

[0133] Table 2 shows the performance comparison on the TJ4DRadset dataset, with the gray area representing the experimental results of this application. Since this application only used single-frame point cloud data in the TJ4DRadset dataset, the point cloud is quite sparse. Therefore, compared to the anchor head method used by PointPillar, which has prior object scale information, C4D-R, which uses a centerhead, naturally performs worse. However, the introduction of image information greatly alleviates the problem of point cloud sparsity, resulting in a significant performance improvement. C4D shows a 16.44% and 14.05% increase in 3D mAP and BEV mAP compared to C4D-R, respectively. Although RCFusion and LXL employ multi-scale sampling strategies, and UniBEVFusion uses swinTransformer as the image encoder, the C4D of this application still outperforms them considerably in both 3D mAP and BEV mAP, showing a 2.41% and 5.08% increase in 3D mAP and BEV mAP compared to the latest UniBEVFusion, respectively. One of the most important reasons is that the Height BEVPool proposed in this application retains the height information of the 3D feature point cloud during the BEV generation process. Combined with PVHG, it can better locate the height of the object, and ultimately effectively alleviate the problem of large differences in the distribution of objects in the height direction.

[0134] Figure 10a and Figure 10b The detection results of C4D-8x and C4D-R on the TJ4DRadset dataset were visualized. Figure 10a It is the projection of the truth box onto the image. Figure 10b The first line shows the detection results for C4D-8x, and the second line shows the detection results for C4D-R. Figure 10b The first, second, and third columns show the detection results for normal scenes, brightly lit scenes, and dark scenes, respectively. It can be seen that C4D-8x can still detect cyclists and vehicles ahead under both bright and dark conditions, demonstrating the model's robustness. It's worth noting that in some scenes, although objects in the foreground are not labeled in the dataset, C4D-8x can still detect them correctly (e.g., the vehicle detected at the furthest point in the second row and third column).

[0135] Table 3 Ablation experiments on the VoD dataset

[0136]

[0137] Table 4 Ablation experiments on the TJ4DRadset dataset

[0138]

[0139] Tables 3 and 4 present the ablation experiment results on the VoD dataset and TJ4DRadset, respectively. The baseline network structure does not use IGF, Height BEVPool, or PVHG methods, but similar to DualBEV, its visual branch still has BEVMask supervision; however, its foreground probability estimation information is solely provided by the BEV image. Adding IGF to the VoD dataset significantly improves cyclist detection performance within the driving area, demonstrating that fusion guidance can focus the image on moving object information, thereby improving moving object detection performance. Furthermore, on the TJ4DRadset dataset, adding IGF increases 3D-mAP and BEV-mAP by 2.28% and 1.95%, respectively, proving that fusion guidance can also enhance the robustness of image foreground guidance, especially when image quality is significantly affected by the environment. After adding Height BEVPool, the model's 3D-mAP improved across the entire labeled region on the VoD dataset and on the TJ4DRadset. However, height prediction was still not accurate enough. Adding PVHG further improved the distribution of feature values ​​in the image's forward-looking feature map along the height direction, making them closer to the object's center height. Height BEVPool then achieved more accurate height prediction, significantly improving model performance. On the VoD dataset, the mAP values ​​for the entire labeled region and the driving region increased by 3.16% and 2.12%, respectively, while on the TJ4DRadset, the 3D-mAP and BEV-mAP increased by 3.92% and 1.79%, respectively. This demonstrates the effectiveness of Height BEVPool and PVHG. Furthermore, adding PVHG to the VoD dataset effectively improved the model's detection capabilities for all categories, especially small objects, across both the labeled and driving regions. This is because small object features are easily interfered with by background features; PVHG guides the model to focus more on foreground features, thus suppressing background features and ultimately improving performance. It is worth noting that after adding PVHG to the TJ4DRadset, although the accuracy of the rider's BEV-mAP did not increase significantly, the 3D-mAP value did increase by 2.5%, further verifying that PVHG helps Height BEVPool to make more accurate height predictions.

[0140] Table 5 Comparison of inference speed between BEVPoolv2 and Height BEVPool

[0141]

[0142] Table 5 shows a comparison of the inference speed of the proposed Height BEVPool and BEVPoolv2. The latency increased by 0.046 ms, which is negligible compared to the overall network latency, but it significantly contributes to the performance improvement. Table 6 shows a comparison of the proposed method applied to LSS-based and sampling-based methods. The sampling-based method achieved 1.18% and 3.76% higher mAP in the entire labeled region and driving region, respectively, compared to the LSS-based method, proving that the sampling-based method is indeed superior, which is consistent with the conclusions of LXL.

[0143] Table 6 compares the LSS perspective transformation method and the Sample perspective transformation method on the VoD dataset.

[0144]

[0145] The VoD dataset provides LiDAR data, so in the previous experiments, this application used LiDAR information to supervise the depth estimation results of the images by default. The TJ4DRadset dataset, however, does not provide LiDAR data, so LiDAR information was not used for depth supervision. Table 7 shows the performance comparison on the VoD dataset with and without LiDAR information for depth supervision. It can be seen that even without depth supervision, the model in this application still outperforms the baseline LXL in both the labeled and driving regions in terms of mAP, while also being significantly faster, demonstrating the effectiveness of the proposed method.

[0146] Table 7 compares the performance of using LiDAR depth supervision on the VoD dataset.

[0147]

[0148] This application proposes a simple, high-speed, and effective 4D millimeter-wave radar and camera fusion 3D object detection method, named C4D. Specifically, this application proposes two plug-and-play modules: PVHG and Height BEVPool. Height BEVPool alleviates the problem of height dimensionality information loss during existing image BEV generation, effectively improving model performance with negligible increase in inference time. PVHG not only assists Height BEVPool in making more accurate height predictions but also guides the model to focus more on foreground objects. Furthermore, IGF uses interactive guidance to suppress 4D millimeter-wave radar point cloud noise and mitigate the impact of complex weather conditions on image features, ultimately achieving better fusion results. IGF also introduces BEV Mask supervision, enabling the model to focus more on foreground objects for improved detection performance. Experimental results on the TJ4DRadset and VoD datasets validate the effectiveness of this method. To the best of our knowledge, it is currently the fastest 4D millimeter-wave radar and camera fusion algorithm on the VoD dataset, while maintaining sufficiently excellent performance.

[0149] C4D provides a foundational framework that can inspire future research in 3D target detection using 4D millimeter-wave radar and camera fusion. Due to its lightweight nature, C4D allows for the integration of many methods validated in other fields to improve overall model performance. Future research may include upsampling image features at multiple scales to generate more accurate image BEV features, exploring more effective feature interaction mechanisms between different modalities, and developing more efficient radar feature extraction methods.

[0150] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of one embodiment, and the modules or processes shown in the drawings are not necessarily essential for implementing this application.

[0151] Those skilled in the art will understand that the modules in the apparatus of the embodiments can be distributed in the apparatus of the embodiments as described in the embodiments, or they can be located in one or more devices different from this embodiment with corresponding changes. The modules of the above embodiments can be combined into one module, or they can be further divided into multiple sub-modules.

[0152] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A fast image-to-4D millimeter-wave radar fusion method for 3D target detection, characterized in that, include: Acquire RGB images from imaging devices, point clouds from millimeter-wave radar, and a trained C4D detection network; Both the RGB image and the point cloud are input into the trained C4D detection network to perform an object detection process and obtain the detection result of the 3D object. The object detection process is as follows: Forward view image features are extracted from the RGB image; the forward view image features are transformed to obtain image BEV features and corresponding BEV height; radar BEV features are extracted from the point cloud; the image BEV features and the radar BEV features are fused into unified BEV features; 3D target detection is performed by combining the BEV height corresponding to the image BEV features and the unified BEV features to obtain the detection result of the 3D target.

2. The method for fast image and 4D millimeter-wave radar fusion 3D target detection according to claim 1, characterized in that, The trained C4D detection network includes an image encoder, a viewpoint transformation module, a radar encoder, an interactive guidance module, and a detection head connected in sequence. The image encoder is used to extract front view image features from the input RGB image; The view transformation module is used to transform the view of the front view image features to obtain the image BEV features and the corresponding BEV height; The radar encoder is used to extract radar BEV features from the input point cloud. The interactive guidance module is used to fuse the image BEV features and the radar BEV features to obtain a unified BEV feature through interactive guidance. The detection head is used to perform 3D target detection by using the BEV height corresponding to the BEV feature of the image and the unified BEV feature to obtain the detection result of the 3D target.

3. The fast image and 4D millimeter-wave radar fusion 3D target detection method according to claim 2, characterized in that, The image encoder includes a backbone network and a neck network connected in sequence; The backbone network extracts multi-scale forward-looking features from the input RGB image, and the neck network fuses the multi-scale forward-looking features to obtain forward-looking image features.

4. The fast image and 4D millimeter-wave radar fusion 3D target detection method according to claim 2, characterized in that, The viewpoint transformation module includes a deep network, a PVHG module, and a height probability sampling module; The deep network predicts the discrete depth distribution of each feature pixel based on the front view image features, and outputs the discrete depth distribution and the front view image features with compressed channel number. The PVHG module uses the front view image features after the number of channels is compressed to predict the front view image heatmap for each category; guided by the front view image heatmap, it combines the front view features after the number of channels is compressed to output the adjusted front view image features. The height probability sampling module, in conjunction with the discrete depth distribution and the intrinsic and extrinsic parameters of the image device, samples the features of preset points in the BEV space from the adjusted front view image features to obtain the image BEV features and the corresponding BEV height.

5. The fast image and 4D millimeter-wave radar fusion 3D target detection method according to claim 4, characterized in that, The height probability sampling module is used for: Set sampling points; By utilizing the intrinsic and extrinsic parameters of the image device, a correspondence is established between sampling points and depth maps and image feature maps, respectively. Using the discrete depth distribution and the adjusted front view image features, the feature value of each valid sampling point is calculated; The feature values ​​of each effective sampling point in each BEV grid are accumulated to obtain the feature of each BEV grid, and the features of all BEV grids are combined to form the BEV feature of the image. For the same channel containing all valid sampling points in each BEV grid, the height of the valid sampling point corresponding to the largest feature value in the radar coordinate system is selected and determined as the BEV height in that channel; the BEV heights of all channels in each BEV grid are combined to form the height of the BEV grid; the heights of all BEV grids are combined to form the BEV height corresponding to the BEV feature of the image.

6. The method for fast image and 4D millimeter-wave radar fusion 3D target detection according to claim 2, characterized in that, The radar encoder is used to perform volumetric columnarization on the input point cloud, and then encode each non-empty volumetric column to obtain pseudo-image features of the point cloud. Multi-scale features are extracted from the pseudo-image features, and the multi-scale features are fused to obtain the BEV features of the 4D millimeter-wave radar, which are then used as the radar BEV features.

7. The method for fast image and 4D millimeter-wave radar fusion 3D target detection according to claim 2, characterized in that, The interactive guidance module includes: a RAG module, an IGM module, and a splicing module; The RAG module predicts the occupancy probability of the BEV network using image BEV features and radar BEV features, and uses the occupancy probability to guide the image BEV features, resulting in image BEV features that have been jointly guided and adjusted by image BEV features and radar BEV features. The IGM module uses the image BEV features to guide the radar BEV features, thereby obtaining radar BEV features guided by the image BEV features. The stitching module stitches and convolves the guided image BEV features with the guided radar BEV features to obtain unified BEV features.

8. The method for fast image and 4D millimeter-wave radar fusion 3D target detection according to claim 2, characterized in that, The detection head is used to perform convolution, normalization, and nonlinear processing on the unified BEV features to obtain intermediate BEV features; adjust the number of channels of the BEV height corresponding to the BEV features of the image to obtain BEV height features with adjusted channel number; concatenate the BEV height with adjusted channel number and the intermediate BEV features along the channel dimension, and adjust the channel dimension again after concatenation to obtain the final height feature; and use the final height feature and the intermediate BEV features to predict the multi-dimensional information of the prediction box labeled with the 3D target location.

9. The method for fast image and 4D millimeter-wave radar fusion 3D target detection according to claim 8, characterized in that, The multi-dimensional information includes the geometric center coordinates, length, width, height, and orientation angle.

10. A 3D target detection device that fuses rapid imagery with 4D millimeter-wave radar, characterized in that, Includes an acquisition module and a detection module; The acquisition module is configured to acquire RGB images from an image device, point clouds from a millimeter-wave radar, and a trained C4D detection network. The detection module is configured to input both the RGB image and the point cloud into the trained C4D detection network, so that it performs a target detection process to obtain the detection result of the 3D target. The target detection process is as follows: Forward view image features are extracted from the RGB image; the forward view image features are transformed to obtain image BEV features and corresponding BEV height; radar BEV features are extracted from the point cloud; the image BEV features and the radar BEV features are fused into unified BEV features; 3D target detection is performed by combining the BEV height corresponding to the image BEV features and the unified BEV features to obtain the detection result of the 3D target.

Images