A 3D target detection method for 4D radar cameras using Gaussian-enhanced multi-stage fusion

Abstract

This invention discloses a 4D radar camera 3D target detection method using a point-Gaussian enhanced multi-stage fusion approach, belonging to the field of autonomous driving and intelligent vehicle environmental perception technology. The purpose of this invention is to propose a 4D radar camera 3D target detection method that can fully mine the geometric information of radar point clouds and effectively utilize image semantic information for 3D target detection while ensuring real-time detection efficiency. This invention employs 3D Gaussian splashing technology to render the Gaussian primitives onto a bird's-eye view plane, generating radar BEV features. Then, multi-scale pyramid image features are acquired, and Focal Loss is used for supervised training of the foreground scores. These foreground scores are then used as attention weights to multiply back into voxel features, obtaining reweighted fused features, and finally, a fused BEV feature map. This invention effectively solves the problems of feature sparsity and quantization error in traditional cylindrical encoders, avoids the computational bottleneck caused by complex image view transformations, and meets real-time requirements while ensuring detection accuracy, making it suitable for vehicle deployment.

Description

Technical Field

[0001] This invention belongs to the field of autonomous driving and intelligent vehicle environmental perception technology. Background Technology

[0002] With the rapid development of autonomous driving and intelligent transportation technologies, vehicle environmental perception systems have become a key foundation for achieving safe driving and intelligent decision-making. Three-dimensional object detection, as a core task of environmental perception, aims to obtain the category and three-dimensional bounding box information of objects in a scene from sensor data. Currently, autonomous vehicle perception systems typically rely on multiple sensors such as cameras, LiDAR, and millimeter-wave radar.

[0003] LiDAR can acquire high-precision 3D scene point clouds with excellent detection performance. However, LiDAR is expensive and susceptible to adverse weather conditions such as rain and fog. Cameras can provide high-resolution visual information and rich semantic features, but their imaging quality is highly dependent on lighting and weather conditions. 4D millimeter-wave radar has higher resolution and can provide accurate elevation angle measurements compared to traditional 3D radar, while also offering advantages such as low cost, less susceptibility to rain and fog, and long detection range. However, 4D radar point clouds still suffer from sparsity and noise issues, limiting the performance of 3D target detection when relying solely on 4D radar point clouds. Therefore, fusing 4D radar with cameras can provide a cost-effective and robust perception solution.

[0004] Most existing 4D radar-camera fusion methods follow the BEV domain fusion framework designed for LiDAR-camera fusion. Some studies use orthogonal feature transformation to obtain image BEV feature maps and design interactive attention modules for fusion; other studies use depth-based sampling strategies to upscale image features to 3D space; still others propose a multi-stage voxel-image fusion strategy to achieve deep interaction between point cloud features and image features.

[0005] However, the above method has the following shortcomings: First, radar point cloud feature extraction is insufficient. Most existing methods use cylinder encoders or voxel encoders designed for dense lidar point clouds, without fully considering the extremely sparse characteristics of 4D radar point clouds. Cylinder encoders extract features from each non-empty cylinder, resulting in extremely sparse feature maps, and quantization errors during cylinderization can lead to the loss of fine geometric information of the target. Secondly, during the voxel-image feature fusion process, due to the sparse radar point cloud, the position of the voxel center projected onto the image may be biased, resulting in a mismatch between the sampled image features and the actual spatial position of the voxels, which affects the fusion effect. Third, it is difficult to balance detection accuracy and real-time performance. Existing methods involve complex image view transformations and multimodal fusion modules that require processing a large number of dense feature maps, resulting in huge computational overhead and limiting their deployment on automotive embedded platforms. Summary of the Invention

[0006] The purpose of this invention is to propose a 4D radar camera 3D target detection method that can fully mine radar point cloud geometric information and effectively utilize image semantic information for 3D target detection while ensuring real-time detection efficiency. This method employs a point Gaussian enhancement multi-stage fusion approach.

[0007] The steps of this invention are: S1. Acquisition of point Gaussian encoder: First, acquire 4D radar point cloud data, then extract point features containing spatial location, radar cross section and radial velocity information for each radar point, input the point features of each radar point into the attribute prediction network to predict the attribute parameters of the corresponding Gaussian element, and finally, use three-dimensional Gaussian splashing technology to render the Gaussian element onto the bird's-eye view plane to generate radar BEV features. S2. Image Branch Acquisition: First, acquire camera image data. Use a feature pyramid network as the neck network, and fuse multi-scale features through top-down and lateral connections to obtain multi-scale pyramid image features. ,in, Indicates the first Multi-scale image features This represents the total number of levels in the feature pyramid. This is the hierarchical index number of the feature pyramid; S3. Multi-stage voxel-image fusion backbone: The multi-stage voxel-image fusion backbone consists of n fusion blocks and m regular blocks cascaded together. Each regular block comprises a sparse convolutional layer and several sub-manifold convolutional layers or residual blocks. The backbone fusion employs a multi-scale deformable feature fusion method, as detailed below: For the first fusion block, initialize the query vector. ,in Features sampled from the first-level image feature map, For the normalized center coordinates of the voxels; for subsequent fusion blocks, Use it directly as the query vector; query vector The sampling offset is obtained through two parallel linear layers. and weight ,in , This represents the total number of levels in the feature pyramid. The number of sampling points for each feature map. Index of sampling points within the same level; Image features are obtained through weighted summation: (8) in, Indicates the first Level feature map Normalized attention weights for each sampling point, , For the weights of deformable attention prediction, Indicates the first Level feature map The location of each sampling point, and , Represents the normalized coordinates of the body's center of mass projected onto the image plane. For deformable attention prediction of the first Level feature map Normalized offset of each sampling point; S4. Design of the semantic guidance head: For each non-empty voxel output by the last fusion block, its fusion features are passed through a multilayer perceptron to obtain a foreground score. Focal Loss is used to supervise the training of the foreground score. The foreground score is used as an attention weight to multiply back the voxel feature to obtain the reweighted fusion feature. S5 and the three-dimensional neck aggregate the multi-scale features output by the last few modules to obtain a fused BEV feature map.

[0008] The 4D radar point cloud data described in step S1 of this invention: Get Time 4D radar point cloud data, point set as ,in This represents the number of radar point clouds in that frame. The radar points are indexed and numbered from 1 to... , indicating the first Each radar point Including spatial location radial velocity With radar cross section .

[0009] The extracted point features described in step S1 of this invention are: (1) in, Indicates radar point characteristics, This represents a multilayer perceptron network. Represents the three-dimensional spatial coordinates of the radar point. Indicates the radar cross section. Indicates radial velocity.

[0010] The attribute parameters of the Gaussian elements corresponding to the prediction points mentioned in step S1 of this invention: Each Gaussian cell contains a mean position offset. Three-dimensional scale Rotation Quaternions and opacity ; The position of the Gaussian element in three-dimensional space is: (2) in, This indicates the position of the Gaussian element in three-dimensional space. Indicates the spatial coordinates of the radar point; This indicates the shift in the predicted mean position. Represents the real number field, which is a one-dimensional real number. Representing three-dimensional real space, Represents the four-dimensional real number space; The covariance matrix of Gaussian elements is calculated using three-dimensional scaling and rotational quaternions: (3) in, Represents the covariance matrix of Gaussian elements; Represents the rotation matrix corresponding to the rotation quaternion; Represents a scaled diagonal matrix. , It is a three-dimensional scale vector.

[0011] In the multi-scale pyramid image features described in step S2 of this invention , , and The first The height, width, and number of channels of the level feature map. express The dimensional real space, i.e. the 3rd The actual dimension of the image feature map tensor.

[0012] The processing steps of the fusion block described in step S3 of this invention: 1) Sparse convolution: For input sparse tensors... First, downsampling is performed through a sparse convolutional layer to obtain... ; 2) Physical fitness calculation and projection: calculation The center of mass of each non-empty voxel : (4) in, Indicates the first The centroid coordinates of a non-empty voxel Indicates the location of voxels The collection of radar points within, Indicates the first element in the set. The spatial coordinates of each radar point This represents the local index number within the set of radar points in this voxel, i.e., from 1 to the total number of radar points within this voxel. Represents a set The base number, i.e., voxels The number of internal radar points.

[0013] The multi-stage voxel-image fusion described in step S3 of this invention employs a method of fusion by adding voxel features after stitching and linear mapping. Features are sampled from feature maps at various scales using bilinear interpolation, and then fused with voxel features after concatenation and linear mapping. (6) in, This represents the image features obtained from sampling. This represents a linear mapping layer with batch normalization. This indicates a channel splicing operation. This indicates a bilinear interpolation sampling operation. This represents the total number of levels in the image feature pyramid. Voxel characteristics after fusion for: (7) in, Indicates the characteristics of the original voxel.

[0014] The reweighted fusion feature described in step S4 of this invention is as follows: (10) in, This represents the fusion characteristics after reweighting; This indicates the fusion characteristics before reweighting; This represents the foreground score.

[0015] The present invention implements the point Gaussian enhancement multi-stage fusion 4D radar camera three-dimensional target detection system according to any one of claims 1-8, comprising: A point Gaussian encoding module is used to acquire 4D radar point cloud data; Image feature extraction module, used to acquire camera image data; The multi-stage voxel-image fusion backbone module is used to perform deep interactive fusion of radar voxel features and multi-scale image features through cascaded n fusion blocks and m regular blocks; The semantic guidance module is used for each non-empty voxel output by the last fusion block; The feature aggregation module is used to aggregate the multi-scale features output by the last few modules. The object detection module is used to output the object detection category and 3D bounding box based on the fused BEV feature map, thus obtaining the 3D object detection result.

[0016] The advantages of this invention compared to the prior art are: 1. Highly efficient and comprehensive radar feature extraction: Each point in the sparse radar point cloud is expanded into a Gaussian element by a point Gaussian encoder, and a dense BEV feature map is generated by three-dimensional Gaussian splashing technology, which effectively solves the problems of feature sparsity and quantization error in traditional cylindrical encoders. 2. Multi-stage deep fusion: Deep interaction between point cloud voxel features and multi-scale semantic features of images is achieved through cascaded fusion blocks. The multi-scale deformable feature fusion method can adaptively determine the sampling position and weight according to the query, further enhancing the fusion capability. 3. Effectively improves detection accuracy: The semantic guidance head guides the network to accurately identify 3D foreground points through explicit foreground-background segmentation supervision and feature reweighting, effectively improving detection accuracy; 4. Excellent in both accuracy and real-time performance: It avoids the computational bottleneck caused by complex image view transformations, and meets the real-time requirements while ensuring detection accuracy, making it suitable for vehicle deployment. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of the system architecture of the 4D radar camera three-dimensional target detection method with Gaussian enhancement and multi-stage fusion according to the present invention; Figure 2 This is a schematic diagram of the visualization of BEV feature map and detection results in a typical scenario using the method of the present invention; Figure 3 This is a comparison chart of the detection accuracy of the method of the present invention on different categories on the VoD dataset; Figure 4 This is a schematic diagram comparing the detection results of the method of the present invention with the true values. Detailed Implementation

[0018] The steps of this invention are: S1. Acquisition of point Gaussian encoder: First, acquire 4D radar point cloud data, then extract point features containing spatial location, radar cross section and radial velocity information for each radar point, input the point features of each radar point into the attribute prediction network to predict the attribute parameters of the corresponding Gaussian element, and finally, use three-dimensional Gaussian splashing technology to render the Gaussian element onto the bird's-eye view plane to generate radar BEV features. S2. Image branch acquisition: First, camera image data is acquired. A feature pyramid network is used as the neck network. Multi-scale features are fused through top-down and lateral connections to obtain multi-scale pyramid image features. S3, Multi-stage voxel-image fusion backbone: The multi-stage voxel-image fusion backbone consists of n fusion blocks and m regular blocks cascaded together. The regular blocks are composed of sparse convolutional layers and several submanifold convolutional layers or residual blocks. S4. Design of the semantic guidance head: For each non-empty voxel output by the last fusion block, its fusion features are passed through a multilayer perceptron to obtain a foreground score. Focal Loss is used to supervise the training of the foreground score. The foreground score is used as an attention weight to multiply back the voxel feature to obtain the reweighted fusion feature. S5 and the three-dimensional neck aggregate the multi-scale features output by the last few modules to obtain a fused BEV feature map.

[0019] This invention discloses a 4D radar camera 3D target detection method based on point Gaussian enhancement and multi-stage fusion. The method includes a point Gaussian encoder, image branches, a multi-stage voxel-image fusion backbone, a semantic guidance head, and a target detection head. The invention is further described in detail below. This invention is a 4D radar camera three-dimensional target detection method with point Gaussian enhancement multi-stage fusion. The method includes a point Gaussian encoder, an image branch, a multi-stage voxel-image fusion backbone, a semantic guidance head, and a target detection head.

[0020] The method of the present invention includes the following steps: Step 1: Point Gaussian encoder Step 1.1: 4D Radar Data Acquisition Get Time 4D radar point cloud data, represented as a set of points ,in This represents the number of radar point clouds in that frame. The radar points are indexed and numbered from 1 to... , indicating the first One radar point. Each radar point Including spatial location radial velocity With radar cross section .

[0021] Step 1.2, Point Feature Extraction For each radar point, point features containing spatial location, radar cross section, and radial velocity information are extracted, as shown in Equation (1): (1) in, Indicates radar point characteristics; Represents a multilayer perceptron network; Represents the three-dimensional spatial coordinates of the radar point; Indicates the radar cross section; Indicates radial velocity.

[0022] Step 1.3, Gaussian Meta-Attribute Prediction Point features of each radar point Input an attribute prediction network (multilayer perceptron) to predict the attribute parameters of the corresponding Gaussian elements at a given point. Each Gaussian element contains the predicted mean position offset. 3D scale vector Rotation Quaternions and opacity , Represents the real number field, which is a one-dimensional real number. Representing three-dimensional real space, It represents the four-dimensional real space.

[0023] The position of the Gaussian element in three-dimensional space is shown in equation (2): (2) in This indicates the position of the Gaussian element in three-dimensional space; Indicates the spatial coordinates of the radar point; This indicates the predicted mean position offset.

[0024] The covariance matrix of Gaussian elements is calculated using three-dimensional scaling and rotational quaternions, as shown in equation (3): (3) in, Represents the covariance matrix of Gaussian elements; Represents the rotation matrix corresponding to the rotation quaternion; Represents a scaled diagonal matrix. , It is a three-dimensional scale vector.

[0025] Step 1.4: Generating radar BEV features using 3D Gaussian splashing A 3D Gaussian sputtering technique is used to render the Gaussian elements onto a bird's-eye view plane to generate radar BEV features. Unlike traditional cylindrical encoders, in the Gaussian sputtering process, the Gaussian elements corresponding to each radar point can simultaneously contribute to multiple adjacent BEV grid cells, thereby effectively reducing quantization errors and generating denser and more accurate radar BEV features.

[0026] Specifically, for each grid position on the BEV plane By analyzing the characteristic contributions of all Gaussian elements at this location The raster's eigenvalues ​​are obtained by blending (i.e., weighted summation based on opacity).

[0027] Step 2, Image Branch Step 2.1, Image Acquisition Get Time Camera image data ,in , These represent the height and width of the image, respectively.

[0028] Step 2.2: Multi-scale image feature extraction A residual network is used as the backbone network. Multi-scale image feature maps are extracted through layer-by-layer convolution and downsampling, which contain shallow texture information and deep semantic information.

[0029] A feature pyramid network is used as the neck network, and multi-scale features are fused through top-down and lateral connections to obtain multi-scale pyramid image features. ,in , , and The first The height, width, and number of channels of the level feature map. express The dimensional real space, i.e. the 3rd The actual dimension of the image feature map tensor. This represents the total number of levels in the feature pyramid.

[0030] Step 3: Multi-stage voxel-image fusion backbone Step 3.1, Voxelization The 4D radar point cloud is voxelized to obtain a sparse tensor. ,in express Characteristics of a non-empty voxel Representing the coordinates of these voxels, The number of channels for voxel features.

[0031] Step 3.2: Cascaded Fusion Block and Regular Block The multi-stage voxel-image fusion backbone consists of n fusion blocks and m regular blocks cascaded together, where n and m are both positive integers. Each regular block consists of a sparse convolutional layer and several submanifold convolutional layers or residual blocks.

[0032] Step 3.3, Fusion Block Each fusion block contains the following processing steps: (1) Sparse convolution: For input sparse tensors First, downsampling is performed through a sparse convolutional layer to obtain... .

[0033] (2) Physical fitness calculation and projection: calculation The center of mass of each non-empty voxel As shown in equation (4): (4) in, Indicates the first The coordinates of the centroid of a non-empty voxel; Indicates the location of voxels The set of radar points within; Indicates the first element in the set. The spatial coordinates of each radar point This indicates the local index number (from 1 to the total number of radar points within the voxel) in the set of radar points within the voxel. Represents a set The base number, i.e., voxels The number of internal radar points. For non-empty voxels that do not contain radar points (generated by the dilation effect of sparse convolution), voxel centers are used instead.

[0034] Based on the camera intrinsic parameter matrix The coordinate transformation matrix from radar to camera The body mass is projected onto the image plane, as shown in equation (5): (5) in, Represents the homogeneous coordinates of the mass center projected onto the image plane; Represents the camera intrinsic parameter matrix; This represents the coordinate transformation matrix from radar to camera; Indicates physical fitness and mental health Homogeneous coordinates.

[0035] After projection ,in For pixel coordinates, This corresponds to the depth value. The normalized coordinates are... .

[0036] (3) Image feature sampling and fusion: based on normalized coordinates Image features are sampled from the multi-scale image feature map and fused with voxel features.

[0037] Features are sampled from feature maps at various scales using bilinear interpolation, and then added to and fused with voxel features after concatenation and linear mapping, as shown in equations (6) and (7): (6) in, This represents the image features obtained from sampling; This represents a linear mapping layer with batch normalization; Indicates a channel splicing operation; This indicates a bilinear interpolation sampling operation; Indicates the first Multi-scale image features; Represents the normalized coordinates of the body's center of mass projected onto the image plane; This represents the total number of levels in the image feature pyramid. The hierarchical index number of the feature pyramid (from 1 to ...) ).

[0038] The characteristics of the fused voxels are as follows: (7) in, Indicates the characteristics of the fused voxels; Indicates the characteristics of the original voxel.

[0039] Alternatively, a multi-scale deformable feature fusion approach can be adopted: based on a deformable cross-attention mechanism, the sampling position and weights are adaptively determined according to the query vector. For the first fusion block, the query vector is initialized. ,in Features sampled from the first-level image feature map, For the normalized center coordinates of the voxels; for subsequent fusion blocks, Use it directly as a query vector.

[0040] query vector The sampling offset is obtained through two parallel linear layers. and weight ,in , This represents the total number of levels in the feature pyramid. The number of sampling points for each feature map. For the hierarchical index of the feature pyramid (level 1 to level 2) class), Index of sampling points within the same level (from the 1st to the 2nd) (Number of samples). Sampling locations are... ,in This is the normalized offset.

[0041] Image features are obtained through weighted summation. The multi-scale deformable feature fusion method is shown in Equation (8): (8) in, This represents the image features obtained from sampling; This represents a linear mapping layer with batch normalization; This represents the total number of levels in the image feature pyramid. This represents the number of sampling points for each feature map; Indicates the first Level feature map Normalized attention weights for each sampling point; This indicates a bilinear interpolation sampling operation; Indicates the first Multi-scale image features; Indicates the first Level feature map The location of each sampling point, and ,in For the normalized coordinates of the physical and mental projection, For deformable attention prediction of the first Level feature map Normalized offset of each sampling point , The weights are for deformable attention prediction.

[0042] (4) Residual processing: merging features Further processing via residual blocks results in the output. .

[0043] By cascading multiple fusion blocks, point cloud features and image features achieve multi-stage deep interaction.

[0044] Step 4, Semantic Guide Header For each non-empty voxel output by the last fusion block, its fusion features are... The foreground score is obtained through a multilayer perceptron, as shown in equation (9): (9) in, Represents the foreground fraction of a non-empty voxel; express Activation function; Represents a multilayer perceptron network; This indicates the fusion feature.

[0045] Focal Loss is used for supervised training of foreground scores.

[0046] The foreground score is used as the attention weight and multiplied back into the voxel feature to obtain the reweighted fusion feature, as shown in Equation (10): (10) in, This represents the fusion characteristics after reweighting; This indicates the fusion characteristics before reweighting; This represents the foreground score.

[0047] Step 5: Three-dimensional neck and detection head The 3D neck layer aggregates the multi-scale sparse features output by the last few modules in the backbone network to generate a BEV feature map with uniform resolution. Since the feature maps output by different modules have different spatial resolutions and semantic abstraction levels (shallow modules retain finer-grained spatial location information, while deeper modules have richer semantic information), directly using features from a single layer for detection would result in incomplete information. The 3D neck layer uses the voxel coordinates of the 4th module as a reference and aligns the voxel coordinates of the 5th and 6th modules according to their corresponding downsampling ratios, establishing a spatial correspondence between the features of each layer. Then, all aligned voxel features are merged, and voxel features with the same X and Y coordinates are added together. This preserves multi-scale structural information and avoids the problem of small target loss caused by simply using the highest-level features, ultimately outputting a fused BEV feature map. The specific fusion method for image features and voxel features within the fusion block can be either SFF (Simple Feature Fusion) or MSDFF (Multi-Scale Deformable Feature Fusion). The difference between the two methods is that SFF samples at fixed positions through bilinear interpolation, which is simple to implement and has low computational cost, making it suitable for deployment environments with high real-time requirements or limited computing resources. MSDFF adaptively determines the sampling position and weights through deformable cross-attention, resulting in stronger fusion capabilities and more accurate positioning. It is suitable for complex urban traffic scenarios with dense targets, severe occlusion, and many small targets. In practical applications, the choice can be made based on accuracy and real-time requirements.

[0048] The detection head predicts the 3D bounding box (center position, size, and orientation) and category of objects in the scene based on the fused BEV feature map.

[0049] Step 6, Loss Function The total loss function of this invention is shown in equation (11): (11) in, Indicates the total loss; This represents the segmentation loss of the semantic guide header; This indicates the detection loss of the target detection head; , These are the balancing weight coefficients for segmentation loss and detection loss, respectively.

[0050] This invention relates to a 4D radar camera 3D target detection system based on point Gaussian enhancement and multi-stage fusion, characterized in that it comprises: The point Gaussian encoding module is used to acquire 4D radar point cloud data. It extracts point features containing spatial location, radar cross section and radial velocity information for each radar point. The point features of each radar point are input into the attribute prediction network to predict the attribute parameters of the corresponding Gaussian element. The Gaussian element is rendered onto the bird's-eye view plane using three-dimensional Gaussian splashing technology to generate radar BEV features. The image feature extraction module is used to acquire camera image data. It uses a feature pyramid network as the neck network and fuses multi-scale features through top-down and lateral connections to obtain multi-scale pyramid image features. The multi-stage voxel-image fusion backbone module is used to perform deep interactive fusion of radar voxel features and multi-scale image features through cascaded n fusion blocks and m regular blocks to obtain a fused BEV feature map. The semantic guidance module is used to obtain a foreground score for each non-empty voxel output by the last fusion block through a multilayer perceptron, use Focal Loss to supervise the training of the foreground score, and use the foreground score as an attention weight to multiply back the voxel feature to obtain the reweighted fusion feature. The feature aggregation module is used to aggregate the multi-scale features output by the last few modules to obtain a fused BEV feature map. The object detection module is used to output the object detection category and 3D bounding box based on the fused BEV feature map, thus obtaining the 3D object detection result.

[0051] Dataset Validation To verify the effectiveness of the proposed point Gaussian-enhanced multi-stage fusion method for 3D target detection by 4D radar cameras, a verification experiment was conducted on the VoD (View-of-Delft) dataset.

[0052] 1. Preliminary preparations The experiments were conducted using the MMDetection3D framework. The VoD dataset was collected in Delft, Netherlands, and covers complex urban traffic scenes, containing approximately 8600 frames of synchronized 4D radar, 64-line LiDAR, stereo camera, and vehicle odometer data. The 4D radar was a ZF FRGen21 3+1D radar (approximately 13Hz), mounted behind the front bumper of the vehicle; the stereo camera image resolution was 1936×1216 pixels (approximately 30Hz), mounted on the windshield; and the LiDAR was a Velodyne HDL-64 S3 (approximately 10Hz), mounted on the roof of the vehicle. The dataset contains over 123,000 3D bounding box annotations, covering 13 categories of road users, including cars, pedestrians, and cyclists. Following the convention of existing research, five frames of accumulated 4D radar point clouds were used as input to increase point cloud density and improve detection performance, considering the three categories of cars, pedestrians, and cyclists.

[0053] The image branch was initialized using an HTC network pre-trained on the COCO dataset and fine-tuned on the nuImage dataset. Image branch parameters were frozen during training to maintain the stability of image features. Point cloud voxel sizes were set to 0.05m, 0.05m, and 0.125m along the X, Y, and Z axes, respectively. End-to-end training was performed using the AdamW optimizer with an initial learning rate of 0.001 and a single-cycle learning rate scheduling strategy. Data augmentation strategies such as random flipping, random scaling, and random rotation were employed during training to improve the model's generalization ability.

[0054] 2. Experimental Procedure The trained model was evaluated on the VoD validation set. The evaluation metric was 3D average accuracy (3D AP), which was calculated for the detection accuracy of three categories: cars, pedestrians, and cyclists, in two evaluation regions: the fully labeled area (EAA) and the driving corridor (DC).

[0055] Figure 2 The invention demonstrates the comparison between the detection results of the method in the present invention and the actual values ​​in a typical scenario. The points are 4D radar point clouds, the dark rectangles are the actual target bounding boxes, and the light rectangles are the 3D bounding boxes predicted by the model. The size of the predicted boxes varies in order of target category (largest for cars, followed by cyclists, and smallest for pedestrians). The predicted boxes match the actual target positions well.

[0056] Figure 3 The paper presents a comparison of the detection accuracy of the proposed method for three target classes—vehicles, pedestrians, and cyclists—under both the EAA (Explicit Area) and DC (Directional Control) evaluation metrics. As shown in the figure, the proposed method achieves competitive detection accuracy in both the fully labeled area (EAA) and the driving corridor (DC) evaluation areas.

[0057] Figure 4 The invention presents the BEV feature map generated by the point Gaussian encoder and the quantitative data of the three-dimensional target detection results, where the magnitude of the BEV propagation intensity indicates the strength of the feature response.

[0058] 3. Conclusion Experimental results show that the method of this invention can effectively generate BEV feature maps representing the distribution of targets in a scene. The point Gaussian encoder effectively solves the problems of feature sparsity and quantization error in traditional cylindrical encoders by expanding sparse radar point clouds into Gaussian elements and using 3D Gaussian sputtering technology to generate dense BEV feature maps. The 3D bounding boxes output by the detection head match the real target positions well, achieving competitive detection accuracy for three types of targets: cars, pedestrians, and cyclists. This verifies the detection accuracy and robustness of the method in complex traffic scenarios, making it suitable for deployment in practical autonomous driving scenarios.

Claims

1. A method for 3D target detection using a 4D radar camera with point Gaussian enhancement and multi-stage fusion, characterized in that: The steps are as follows: S1. Acquisition of point Gaussian encoder: First, acquire 4D radar point cloud data, then extract point features containing spatial location, radar cross section and radial velocity information for each radar point, input the point features of each radar point into the attribute prediction network to predict the attribute parameters of the corresponding Gaussian element, and finally, use three-dimensional Gaussian splashing technology to render the Gaussian element onto the bird's-eye view plane to generate radar BEV features. S2. Image Branch Acquisition: First, acquire camera image data. Use a feature pyramid network as the neck network, and fuse multi-scale features through top-down and lateral connections to obtain multi-scale pyramid image features. ,in, Indicates the first Multi-scale image features This represents the total number of levels in the feature pyramid. This is the hierarchical index number of the feature pyramid; S3. Multi-stage voxel-image fusion backbone: The multi-stage voxel-image fusion backbone consists of n fusion blocks and m regular blocks cascaded together. Each regular block comprises a sparse convolutional layer and several sub-manifold convolutional layers or residual blocks. The backbone fusion employs a multi-scale deformable feature fusion method, as detailed below: For the first fusion block, initialize the query vector. ,in Features sampled from the first-level image feature map, For the normalized center coordinates of the voxels; for subsequent fusion blocks, Use it directly as the query vector; query vector The sampling offset is obtained through two parallel linear layers. and weight ,in , This represents the total number of levels in the feature pyramid. The number of sampling points for each feature map. Index of sampling points within the same level; Image features are obtained through weighted summation: （8） in, Indicates the first Level feature map Normalized attention weights for each sampling point, , For the weights of deformable attention prediction, Indicates the first Level feature map The location of each sampling point, and , Represents the normalized coordinates of the body's center of mass projected onto the image plane. For deformable attention prediction of the first Level feature map Normalized offset of each sampling point; S4. Design of the semantic guidance head: For each non-empty voxel output by the last fusion block, its fusion features are passed through a multilayer perceptron to obtain a foreground score. Focal Loss is used to supervise the training of the foreground score. The foreground score is used as an attention weight to multiply back the voxel feature to obtain the reweighted fusion feature. S5 and the three-dimensional neck aggregate the multi-scale features output by the last few modules to obtain a fused BEV feature map.

2. The 4D radar camera three-dimensional target detection method with point Gaussian enhancement multi-stage fusion as described in claim 1, characterized in that: The 4D radar point cloud data mentioned in step S1: Get Time 4D radar point cloud data, point set as ,in This represents the number of radar point clouds in that frame. The radar points are indexed and numbered from 1 to... , indicating the first Each radar point Including spatial location radial velocity With radar cross section .

3. The 4D radar camera three-dimensional target detection method with point Gaussian enhancement multi-stage fusion as described in claim 1, characterized in that: The extracted point features described in step S1 are: （1） in, Indicates radar point characteristics, This represents a multilayer perceptron network. Represents the three-dimensional spatial coordinates of the radar point. Indicates the radar cross section. Indicates radial velocity.

4. The 4D radar camera three-dimensional target detection method with point Gaussian enhancement multi-stage fusion as described in claim 1, characterized in that: The attribute parameters of the Gaussian elements corresponding to the predicted points mentioned in step S1: Each Gaussian cell contains a mean position offset. Three-dimensional scale Rotation Quaternions and opacity ; The position of the Gaussian element in three-dimensional space is: （2） in, This indicates the position of the Gaussian element in three-dimensional space. Indicates the spatial coordinates of the radar point; This indicates the shift in the predicted mean position. Represents the real number field, which is a one-dimensional real number. Representing three-dimensional real space, Represents the four-dimensional real number space; The covariance matrix of Gaussian elements is calculated using three-dimensional scaling and rotational quaternions: （3） in, Represents the covariance matrix of Gaussian elements; Represents the rotation matrix corresponding to the rotation quaternion; Represents a scaled diagonal matrix. , It is a three-dimensional scale vector.

5. The 4D radar camera three-dimensional target detection method with point Gaussian enhancement multi-stage fusion as described in claim 1, characterized in that: In the multi-scale pyramid image features described in step S2 , , and The first The height, width, and number of channels of the level feature map. express The dimensional real space, i.e. the 3rd The actual dimension of the image feature map tensor.

6. The 4D radar camera three-dimensional target detection method based on point Gaussian enhancement multi-stage fusion according to claim 1, characterized in that: Step S3 describes the processing steps for the fusion block: 1) Sparse convolution: For input sparse tensors... First, downsampling is performed through a sparse convolutional layer to obtain... ; 2) Physical fitness calculation and projection: calculation The center of mass of each non-empty voxel : （4） in, Indicates the first The centroid coordinates of a non-empty voxel Indicates the location of voxels The collection of radar points within, Indicates the first element in the set. The spatial coordinates of each radar point This represents the local index number within the set of radar points in this voxel, i.e., from 1 to the total number of radar points within this voxel. Represents a set The base number, i.e., voxels The number of internal radar points.

7. The 4D radar camera three-dimensional target detection method based on point Gaussian enhancement and multi-stage fusion according to claim 1, characterized in that: Step S3 describes multi-stage voxel-image fusion: a fusion method combining stitching and linear mapping with voxel features. Features are sampled from feature maps at various scales using bilinear interpolation, and then fused with voxel features after concatenation and linear mapping. （6） in, This represents the image features obtained from sampling. This represents a linear mapping layer with batch normalization. This indicates a channel splicing operation. This indicates a bilinear interpolation sampling operation. This represents the total number of levels in the image feature pyramid. Voxel characteristics after fusion for: （7） in, Indicates the characteristics of the original voxel.

8. The 4D radar camera three-dimensional target detection method with point Gaussian enhancement multi-stage fusion as described in claim 1, characterized in that: The reweighted fusion feature described in step S4 is as follows: （10） in, This represents the fusion characteristics after reweighting; This indicates the fusion characteristics before reweighting; This represents the foreground score.

9. A 4D radar camera three-dimensional target detection system that implements the method of any one of claims 1-8 using point Gaussian enhancement multi-stage fusion, characterized in that: include: A point Gaussian encoding module is used to acquire 4D radar point cloud data; Image feature extraction module, used to acquire camera image data; The multi-stage voxel-image fusion backbone module is used to perform deep interactive fusion of radar voxel features and multi-scale image features through cascaded n fusion blocks and m regular blocks; The semantic guidance module is used for each non-empty voxel output by the last fusion block; The feature aggregation module is used to aggregate the multi-scale features output by the last few modules. The object detection module is used to output the object detection category and 3D bounding box based on the fused BEV feature map, thus obtaining the 3D object detection result.

Images