A 4D radar and vision sensor fusion road target detection method

Abstract

This invention discloses a road target detection method that fuses 4D radar and visual sensors, belonging to the field of intelligent transportation environment perception and autonomous driving technology. First, radar point cloud data and camera image data are acquired and processed to obtain radar BEV features, 3D radar occupancy weights, and image features. Then, the radar BEV features and image features are input into an attention-guided depth distribution supervision module. By combining an attention mechanism with depth distribution supervision, a depth probability distribution map is generated. Next, by combining image features, 3D radar occupancy weights, and the depth probability distribution map, image BEV features are generated. Finally, the radar BEV features and image BEV features are input into a high-efficiency gated fusion module. By combining a difference-driven attention mechanism with gated residual fusion, fused BEV features are generated and input into the detection head, outputting the detection result. This invention can improve the accuracy and system stability of 3D road target detection.

Description

Technical Field

[0001] This invention relates to a road target detection method, specifically a road target detection method that integrates 4D radar and visual sensors, belonging to the field of intelligent transportation environment perception and autonomous driving technology. Background Technology

[0002] Currently, autonomous driving systems rely on high-precision 3D environmental perception technology to detect and locate traffic participants such as vehicles, pedestrians, and cyclists. Existing 3D target detection technologies mainly include pure vision solutions, lidar solutions, and radar-vision fusion solutions. Among them, the radar-vision fusion solution utilizes the physical distance information of millimeter-wave radar and the semantic information of the camera to complement each other, and has the advantages of low cost and strong resistance to severe weather. However, existing technologies still have the following shortcomings: (1) The multimodal fusion method is fixed and cannot dynamically adjust the contribution ratio of different modes according to the scene; (2) Image depth prediction lacks radar geometric constraints, and the error is large at long distances and in occluded areas; (3) Radar point clouds are sparse and contain noise, and direct fusion is prone to introducing errors.

[0003] Therefore, how to construct a three-dimensional target detection method that can achieve adaptive fusion and introduce a radar depth distribution supervision mechanism has become a key problem that urgently needs to be solved in this field. Summary of the Invention

[0004] The purpose of this invention is to provide a three-dimensional target detection method based on the fusion of 4D millimeter-wave radar and vision. By constructing an attention-guided depth distribution supervision module and a difference-driven high-efficiency gating fusion module, this invention solves the problems of static multimodal fusion, unstable depth prediction, and radar noise interference in the prior art, thereby improving the accuracy and stability of three-dimensional target detection.

[0005] To achieve the above objectives, the present invention provides a road target detection method that fuses 4D radar and visual sensors, comprising the following steps:

[0006] Step 1: Acquire radar point cloud data and camera image data;

[0007] Step 2: Extract features from the radar point cloud data and the camera image data respectively to obtain radar BEV features and image features; the radar BEV features are then processed by a 3D radar occupancy network to generate 3D radar occupancy weights.

[0008] Step 3: Input the radar bird's eye view (BEV) features and the image features into the attention-guided depth distribution supervision module, and generate a depth probability distribution map by combining the attention mechanism and depth distribution supervision;

[0009] Step 4: Combine the image features, the 3D radar occupancy weights, and the depth probability distribution map into image BEV features via the View Transformation module;

[0010] Step 5: Input the radar BEV features and the image BEV features into the high-efficiency gated fusion module, and generate fused BEV features by combining the difference-driven attention mechanism with gated residual fusion;

[0011] Step 6: Input the fused BEV features into the detection head for processing and output the detection results.

[0012] Further, step 2 includes:

[0013] The radar point cloud data is input into the radar feature extraction module, which converts it into a sparse voxel mesh through pillarization, and extracts radar BEV features using the SECOND network backbone. The radar BEV features are then processed by the 3D radar occupancy network to generate 3D radar occupancy weights.

[0014] Camera image data is input into the image feature extraction module, and multi-scale image features are extracted using the YOLOX network.

[0015] Furthermore, in step 3, the specific processing procedure of the attention-guided deep distribution supervision module includes:

[0016] Radar BEV features are mapped to key vectors and value vectors, and image features are mapped to query vectors;

[0017] The query vector, key vector, and value vector are input into the attention layer, and the attention mapping features are output through a multi-head cross-attention mechanism.

[0018] Attention-mapped features are restored to the image feature scale via bilinear interpolation to obtain multi-scale features, which are then fused with image features via residual connections to obtain fused multi-modal features.

[0019] The multimodal features are normalized using the softmax function to obtain a depth probability distribution map;

[0020] The loss function is calculated using the error between the predicted depth map obtained from the depth probability distribution map and the sparse depth map obtained from radar point cloud data, and a sparse mask constraint is applied to achieve depth distribution supervision.

[0021] Furthermore, the loss function The calculation formula is:

[0022] ,

[0023] in, To smooth out L1 loss, For the set of valid pixels, The predicted depth value for a pixel. is the sparse depth value of the pixel.

[0024] Furthermore, step five includes:

[0025] First, feature alignment processing is performed on the image BEV features and radar BEV features to obtain image projection features and radar projection features.

[0026] Then, the image projection features and the radar projection features are processed using a difference-driven attention mechanism to obtain image-enhanced BEV features and radar-enhanced BEV features.

[0027] Next, a lightweight weighted prediction network is used to process the image-enhanced BEV features and the radar-enhanced BEV features to obtain fused features;

[0028] Finally, a gated residual fusion mechanism is used to process the projected features and fused features to obtain fused BEV features.

[0029] Furthermore, the implementation process of the difference-driven attention mechanism includes:

[0030] First, the element-wise absolute difference between the image projection features and the radar projection features is calculated to obtain the difference map D;

[0031] Subsequently, a lightweight convolutional network is used to perform a non-linear mapping on the difference map to obtain the attention response map. Specifically:

[0032] ,

[0033] in, For Sigmoid function, Conv is the convolution operation, and ReLU is the ReLU activation function;

[0034] Next, the attention response map is used to perform weighted enhancement on the image projection features and radar projection features, specifically as follows:

[0035] ,

[0036] ,

[0037] in, and These are radar-enhanced BEV features and image-enhanced BEV features, respectively. and These are radar projection features and image projection features, respectively; This is for dot product calculation.

[0038] Furthermore, the implementation process of the gated residual fusion mechanism includes:

[0039] First, the fused features are input into a gating network for processing to generate a gating graph. Specifically:

[0040] ,

[0041] in, Indicates the number of executions; Conv represents the convolution operation; ReLU represents the ReLU activation function; BN represents batch normalization.

[0042] Then, the fused features are processed using the gated graph. Residual fusion is performed with the projected features to obtain the final fused BEV features, specifically:

[0043] ,

[0044] in, As a feature of fusion, To integrate BEV features.

[0045] The present invention also provides an electronic device, including a processor and a memory, the memory storing program code that, when executed by the processor, causes the processor to perform the steps of the method.

[0046] The present invention also provides a storage medium storing a computer program or instructions that, when the computer program or instructions are run on a computer, execute the steps of the method described.

[0047] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0048] 1. Utilizing sparse but physically reliable distance information provided by 4D millimeter-wave radar, a sparse depth supervision signal is generated through projection. An attention-guided mechanism is introduced during depth distribution prediction to impose probabilistic constraints on the visual depth distribution. Compared to existing methods that rely solely on image features for depth prediction, this invention effectively reduces depth estimation errors under long-distance, occluded, and complex lighting conditions by introducing cross-modal geometric priors, thereby significantly improving the accuracy and stability of 3D target localization.

[0049] 2. Constructing an adaptive gating fusion mechanism in the BEV (Bird's-Eye View) space to achieve dynamic synergy of multimodal information. This invention designs a difference-driven gating fusion method in the BEV space. By calculating the difference response between radar features and visual features, it dynamically generates fusion weights to achieve adaptive weighted fusion of multimodal features. Compared with traditional feature stitching or fixed-weight fusion methods, this method can adaptively adjust the contribution ratio of each modality according to different scenarios and target characteristics, suppressing interference in areas with high radar noise and enhancing the role of radar information in areas with degraded visual information, thereby fully leveraging the complementary advantages of the two sensors.

[0050] 3. Improved accuracy of multi-class 3D target detection and enhanced overall perception performance. Based on the aforementioned deep supervision mechanism and adaptive fusion strategy, this invention achieves superior performance in multimodal 3D target detection tasks. Experimental results show that on the View-of-Delft dataset, the method achieves a 3D average accuracy (mAP3D) of 57.87% across the entire labeled region, a 1.56% improvement over the baseline method. It also achieves more significant performance improvements in small target categories such as pedestrians, indicating that this invention has stronger detection capabilities in complex traffic scenarios. Attached Figure Description

[0051] Figure 1 This is a schematic diagram of the method architecture of the present invention;

[0052] Figure 2 This is a schematic diagram of the attention-guided depth distribution supervision module structure of the present invention;

[0053] Figure 3 This is a schematic diagram of the high-efficiency gating fusion module structure of the present invention. Detailed Implementation

[0054] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments, and the advantages of the present invention in the above and / or other aspects will become clearer.

[0055] This invention utilizes the LXL model as the main framework, which is derived from Xiong W, Liu J, Huang T, et al. LXL: LiDAR excluded lean 3D object detection with 4D imaging radar and camerafusion[J].IEEE Transactions on Intelligent Vehicles, 2023, 9(1): 79-92.

[0056] However, the LXL model still has limitations in feature fusion and deep supervision: First, the fusion strategy is relatively static. LXL only performs multimodal fusion by concatenating convolutions, which makes it difficult to simultaneously take into account low-level geometric information and high-level semantic information. Moreover, the fixed fusion method has limited adaptability to complex scenes and fails to fully capture cross-modal semantic dependencies. Second, the depth estimation network of the LXL model only performs depth prediction through explicit and simple image features. However, image depth estimation itself has ill-posedness, and the error is particularly obvious at long distances and in occluded areas, which in turn affects the accuracy of 3D target localization.

[0057] To overcome the aforementioned problems and the accuracy issues in 3D target detection in road scenarios, this invention provides a road target detection method that fuses 4D radar and visual sensors. The overall architecture of this method is as follows: Figure 1 As shown, the specific steps include:

[0058] Step 1: Acquire radar point cloud data and camera image data;

[0059] The method of this invention uses the publicly available View-of-Delft (VoD) dataset for experiments. The dataset contains RGB images from a monocular camera and point cloud data from a 4D radar, and is labeled with 3D bounding boxes.

[0060] In the preprocessing stage, radar point clouds and camera images are standardized and then invalid data is removed by filtering through regions of interest.

[0061] Step 2: Extract features from radar point cloud data and camera image data respectively to obtain radar BEV features, 3D radar occupancy weights and image features;

[0062] The radar feature extraction module converts the radar point cloud into a sparse voxel mesh through pillarization. The radar BEV features are then extracted via the RadarBackbone and RadarNeck networks (specifically, the SECOND network). The radar BEV image is then processed by the Radar 3D Occupancy Net to generate 3D occupancy mesh weights. The image feature extraction module is consistent with LXL, using the ImageBackbone and ImageNeck networks (specifically, the YOLOX network) for feature extraction, generating multi-scale image features. These features are then passed to the attention-guided depth distribution supervision module to generate depth distribution probabilities. The 3D occupancy mesh weights, depth distribution probabilities, and multi-scale image features are converted into image BEV features in the View Transformation module, and finally, together with the radar BEV, they are fed into the EFG module for multimodal fusion.

[0063] The SECOND network used in the radar feature extraction module is derived from: Yan Y, Mao Y, Li B. Second: Sparsely embedded convolutional detection[J]. Sensors, 2018, 18(10):3337.

[0064] The YOLOX network used in the image feature extraction module is derived from: Ge Z, Liu S, Wang F. Yolox: Exceeding yolo series in 2021[J]. arXiv preprint arXiv: 2107.08430, 2021.

[0065] Step 3: Input the radar BEV features and image features into the Attention-Guided Depth Distribution Supervision (ADDS) module. By combining the attention mechanism with depth distribution supervision, a depth probability distribution map is generated.

[0066] The structure of the ADDS module is as follows: Figure 2 As shown, the multi-head cross-attention mechanism is used to capture the spatial correlation between image features and radar features.

[0067] The calculation process of the multi-head cross-attention mechanism is as follows:

[0068] The target feature is represented by the smallest scale feature from the image features (from the image branch), i.e. Feature size is After linear mapping matrix Mapped to query vector By interacting with radar features, the spatial relationship between image features and radar features is learned.

[0069] Key vector K and value vector V are radar features. Feature size is After linear mapping matrix and The result of the transformation is denoted as and .

[0070] By calculating the similarity between Q and K, and then scaling and normalizing it, the attention weights can be obtained. These weights are then used to perform a weighted summation of the values ​​to obtain the final output. The calculation process can be represented as follows:

[0071] ,

[0072] in, To query the dimension items of the vector and key vector, the item This is used to calculate the similarity between each pixel. Finally, Q is residually linked with the final output of the attention function.

[0073] Since memory consumption may increase significantly when processing multi-scale features, the ADDS module uses the smallest scale feature map of the image for initial interactive computation in order to optimize memory usage.

[0074] The computational cost of the smallest-scale feature map is far less than that of the original resolution feature map. This makes attention computation more efficient, avoiding the massive computation required by high-resolution feature maps.

[0075] (1) Image minimum scale feature map interaction: Assume the image feature map size is The size of the minimum scale feature map is Then only for the smallest scale Interactive computation of feature maps will greatly reduce computational load and GPU memory consumption.

[0076] (2) Upsampling Interpolation Recovery: After the interactive computation is completed, the obtained attention map is recovered to a higher resolution feature map using bilinear interpolation. This process ensures information transfer between different scales, ensuring that high-resolution detail information can be effectively transferred to the higher-scale feature map. The formula is expressed as:

[0077] ,

[0078] in, This represents the attention map computed at the smallest scale. This indicates an interpolation operation (such as bilinear interpolation). This represents the attention map restored to its original scale.

[0079] After completing the interpolation recovery, the ADDS module further introduces residual connections to fuse the recovered multi-scale features with the original multi-scale features, specifically: ,

[0080] in, This represents the fused multimodal features. Through this residual approach, the model can preserve the spatial details and structural information of the original features to the greatest extent possible while ensuring cross-modal feature interaction.

[0081] Multimodal features after fusion First, the target depth D is obtained by convolutional channel number, and then the depth probability distribution is obtained by softmax normalization:

[0082] ,

[0083] in It is the first The center depth value of each depth interval; It is a temperature parameter used to control the sharpness of the distribution; For pixels Belongs to the depth range The probability of.

[0084] Therefore, continuous predicted depth can be obtained through the soft-argmax operation: , This is the predicted depth value for that pixel. This operation can achieve a smooth depth prediction while preserving gradient transitivity.

[0085] The loss function is calculated using the error between the predicted depth map obtained from the depth probability distribution map and the sparse depth map obtained from radar point cloud data, and a sparse mask constraint is applied to achieve depth distribution supervision.

[0086] The depth distribution supervision mechanism utilizes sparse depth maps from radar point clouds or 3D box projections as reliable supervision signals.

[0087] The process of generating sparse depth maps and constructing masks is as follows:

[0088] Let the first point in the radar point cloud be... The points are Its projected position in the camera coordinate system is

[0089] ,

[0090] in The extrinsic parameter matrix of the radar to the camera, Represents the perspective projection function. These are pixel coordinates. If multiple points fall on the same pixel... Take the minimum depth value: ,

[0091] in Represents all projections to pixels The point set. This represents the depth value of a pixel. If a pixel has no corresponding point, then it is defined as follows: This indicates that the pixel has no effective supervision. Therefore, the binary mask is defined as:

[0092] ,

[0093] During training only The loss is calculated on the pixels to ensure the stability of sparse supervision. The effective set of pixels is denoted as . .

[0094] The total loss of deep supervision is defined as: ,

[0095] Continuous supervision (SmoothL1) utilizes prediction depth values ​​at the continuous prediction level. With sparse depth value Error calculation and smoothing L1 loss:

[0096] ,

[0097] Sparse mask constraint: loss only The calculation is performed on pixels to avoid the influence of invalid regions. The SmoothL1 term improves the accuracy of continuous depth. , where x is the difference between the predicted depth value and the sparse depth value.

[0098] This invention introduces a sparse but physically reliable distance prior provided by millimeter-wave radar through an Attention-Guided Depth Distribution Supervision (ADDS) module, thereby imposing probabilistic constraints on the visual depth distribution. By combining the attention mechanism with depth distribution supervision, an accurate depth probability distribution is generated. This module enhances the geometric consistency between image and radar features, replacing the original LXL depth estimation network and improving the accuracy of depth estimation.

[0099] Step 4: Combine image features, 3D radar occupancy weights, and depth probability distribution maps to generate image BEV features;

[0100] 3D radar occupancy weights, image features, and depth probability distribution Figure 1 Then, proceed to the ViewTransformation module. Predefined voxels backsample image features to generate 3D voxel features. These voxel features are then multiplied by the 3D radar occupancy weights and depth probability distribution map, respectively. The resulting multiplied features are then summed to obtain the BEV features of the image.

[0101] Step 5: Input the radar BEV features and image BEV features into the EfficientGated Fusion (EFG) module, and generate fused BEV features by combining the difference-driven attention mechanism with gated residual fusion.

[0102] The structure of the high-efficiency gating fusion module is as follows: Figure 3 As shown, it includes a feature alignment layer, a difference-driven attention mechanism, an adaptive fusion mechanism, and a gated residual fusion.

[0103] In the EFG module, the convolutional layers used for feature extraction employ 3×3 convolutional kernels with a stride of 1 and padding of 1.

[0104] The process of feature alignment layer is as follows:

[0105] Due to the BEV characteristics of the image Radar BEV characteristics The distribution space varies, so channel alignment and scale unification are required first. The EFG module uses two independent 1×1 convolutional layers to perform a linear transformation on the input features to obtain the projected features:

[0106] ,

[0107] ,

[0108] This operation ensures that the features of the two modalities can be directly computed and fused within the same semantic space, providing a foundation for subsequent difference modeling.

[0109] The implementation process of the difference-driven attention mechanism is as follows:

[0110] To explicitly model the complementarity between modalities, the EFG module introduces a difference-driven attention mechanism. First, it calculates the element-wise absolute difference between the features of the two modalities to capture the difference regions: D is the difference diagram.

[0111] Subsequently, a lightweight convolutional network is used to perform a non-linear mapping on the difference map to obtain the attention response map. :

[0112] ,

[0113] in, The sigmoid function is used to constrain the attention value to a certain value. Within this range, Conv represents a convolution operation; Relu represents the Relu activation function. This attention map reflects the spatial distribution of the differences between the two modalities, with regions exhibiting greater differences receiving higher weights.

[0114] The attention map is used to perform weighted enhancement on image projection features and radar projection features, as shown below:

[0115] ,

[0116] ,

[0117] in, and These are radar-enhanced BEV features and image-enhanced BEV features, respectively. and These are radar projection features and image projection features, respectively. This is for dot product calculation.

[0118] The adaptive fusion mechanism works as follows:

[0119] The enhanced features require intermodal information integration. The EFG module uses a lightweight weighted prediction network to achieve adaptive fusion. First, the enhanced features of the two modalities are concatenated along the channel dimension: ,

[0120] The concatenated feature maps are then input into a weighted network consisting of two convolutional layers. After ReLU activation and Softmax normalization, the weights for the two modalities are obtained.

[0121] ,

[0122] ,

[0123] Final fusion features for: ,

[0124] in Indicates the number of times it is executed. These represent the weights of the image and radar, respectively; this adaptive weighting method can automatically adjust the contribution of image and point cloud features according to different scenarios, achieving dynamic balance at the information level.

[0125] The gating residual fusion process is as follows:

[0126] To achieve an adaptive balance between enhanced and original features, the EFG module introduces a gated residual mechanism, such as... Figure 3 As shown, the gating network generates a gating graph through two layers of convolution and sigmoid activation. :

[0127] ,

[0128] in This indicates the number of executions; this gating parameter controls the ratio of enhanced features to original features in the final fusion result. The final output features are:

[0129] ,

[0130] The average residual features of the two modalities are used to preserve basic information during the fusion process and prevent over-enhancement from causing feature shift.

[0131] This invention employs a lightweight gating mechanism through an efficient gating fusion module (EFG) to automatically adjust the contribution of image and radar features in the BEV space based on the correlation of input features, thereby achieving dynamic, multimodal feature fusion.

[0132] Step 6: Input the fused BEV features into the detection head for processing and output the detection results.

[0133] To verify the effectiveness of the proposed RXL-Net multimodal 3D target detection method, experimental analysis was conducted focusing on key technical issues in current 4D radar and image fusion perception. Specifically, these include: first, the lack of effective geometric constraints in image branches during depth estimation affects 3D positioning accuracy; and second, existing multimodal fusion methods often employ static fusion strategies, making it difficult to fully utilize complementary information between different modalities.

[0134] To address the aforementioned issues, this invention proposes an Attention-Guided Deep Distribution Supervision Module (ADDS) and an Efficient Gated Fusion Module (EGF), and constructs an RXL-Net model. Experiments are conducted to verify its improved performance in terms of detection accuracy and system robustness.

[0135] Experiments were conducted using the publicly available View-of-Delft (VoD) dataset. The dataset contains RGB images from a monocular camera and point cloud data from a 4D radar, with 3D bounding boxes annotated. During preprocessing, the radar point cloud and camera images were standardized, and invalid data was removed through region of interest filtering. During training, horizontal flipping and point cloud shuffling were used for data augmentation.

[0136] Training settings: RXL-Net uses the AdamW optimizer with an initial learning rate of 0.001 and weight decay of 0.01. The common learning rate follows a cosine annealing strategy, with a warm-up of 1000 steps, and then gradually decays to 1e-5 over the entire 100-batch training process. All experiments are implemented in PyTorch and performed on two NVIDIA RTX A6000 GPUs. Ablation experiments are shown in Table 1.

[0137] Table 1 Ablation Experiment Results

[0138]

[0139] Average Precision (AP) is an evaluation metric for object detection in a single class; mean Average Precision (mAP) is the average of AP across all classes. Bold numbers in the table represent the best detection results. The baseline model (base) corresponds to the LXL framework, and its mAP... 3DThe overall performance was 56.31%. After adding the ADDS module, the model showed improvement in pedestrian and cyclist categories, increasing the overall performance to 56.76%. The introduction of the EFG module resulted in particularly significant improvements in pedestrian and vehicle categories, with mAP... 3D It reached 57.67%.

[0140] When ADDS and EFG are added simultaneously, the model performance is further improved, reaching its optimum in the vehicle and pedestrian categories, and ultimately achieving higher mAP. 3D It reached 57.87%, the best result among all combinations.

[0141] This invention provides a road target detection method that fuses 4D radar and visual sensors. Many methods and approaches exist for implementing this technical solution; the above description is merely a preferred embodiment of the invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this invention, and these improvements and modifications should also be considered within the scope of protection of this invention. All components not explicitly stated in this embodiment can be implemented using existing technologies.

Claims

1. A road target detection method integrating 4D radar and visual sensors, characterized in that, Includes the following steps: Step 1: Acquire radar point cloud data and camera image data; Step 2: Extract features from the radar point cloud data and the camera image data respectively to obtain radar BEV features and image features; The radar BEV features are then processed by a three-dimensional radar occupancy network to generate three-dimensional radar occupancy weights. Step 3: Input the radar BEV features and the image features into the attention-guided depth distribution supervision module, and generate a depth probability distribution map by combining the attention mechanism with depth distribution supervision; Step 4: Combine the image features, the 3D radar occupancy weights, and the depth probability distribution map to generate image BEV features; Step 5: Input the radar BEV features and the image BEV features into the high-efficiency gated fusion module, and generate fused BEV features by combining the difference-driven attention mechanism with gated residual fusion; Step 6: Input the fused BEV features into the detection head for processing and output the detection results.

2. The road target detection method based on the fusion of 4D radar and visual sensors according to claim 1, characterized in that, Step 2 includes: The radar point cloud data is input into the radar feature extraction module, which converts it into a sparse voxel mesh through a cylinder operation. The radar BEV features are then extracted using the radar backbone network and the radar neck network. Finally, the 3D radar occupancy network is used to generate 3D radar occupancy weights. The camera image data is input into the image feature extraction module to extract multi-scale image features.

3. The road target detection method based on the fusion of 4D radar and visual sensors according to claim 1, characterized in that, In step 3, the specific processing procedure of the attention-guided depth distribution supervision module includes: Radar BEV features are mapped to key vectors and value vectors, and image features are mapped to query vectors; The query vector, key vector, and value vector are input into the attention layer, and the attention mapping features are output through a multi-head cross-attention mechanism. Attention-mapped features are restored to the image feature scale via bilinear interpolation to obtain multi-scale features, which are then fused with image features via residual connections to obtain fused multi-modal features. The multimodal features are normalized using the softmax function to obtain a depth probability distribution map; The loss function is calculated using the error between the predicted depth map obtained from the depth probability distribution map and the sparse depth map obtained from radar point cloud data, and a sparse mask constraint is applied to achieve depth distribution supervision.

4. The road target detection method based on the fusion of 4D radar and visual sensors according to claim 3, characterized in that, The loss function The calculation formula is: ， in, To smooth out L1 loss, For the set of valid pixels, The predicted depth value for a pixel. is the sparse depth value of the pixel.

5. The road target detection method based on the fusion of 4D radar and visual sensors according to claim 1, characterized in that, Step five includes: First, feature alignment processing is performed on the image BEV features and radar BEV features to obtain image projection features and radar projection features. Then, the image projection features and the radar projection features are processed using a difference-driven attention mechanism to obtain image-enhanced BEV features and radar-enhanced BEV features. Next, a lightweight weighted prediction network is used to process the image-enhanced BEV features and the radar-enhanced BEV features to obtain fused features; Finally, a gated residual fusion mechanism is used to process the projected features and fused features to obtain fused BEV features.

6. The road target detection method based on the fusion of 4D radar and visual sensors according to claim 5, characterized in that, The implementation process of the difference-driven attention mechanism includes: First, the element-wise absolute difference between the image projection features and the radar projection features is calculated to obtain the difference map D; Subsequently, a lightweight convolutional network is used to perform a non-linear mapping on the difference map to obtain the attention response map. Specifically: ， in, For Sigmoid function, Conv is the convolution operation, and ReLU is the ReLU activation function; Next, the attention response map is used to perform weighted enhancement on the image projection features and radar projection features, specifically as follows: ， ， in, and These are radar-enhanced BEV features and image-enhanced BEV features, respectively. and These are radar projection features and image projection features, respectively; This is for dot product calculation.

7. The road target detection method based on the fusion of 4D radar and visual sensors according to claim 5, characterized in that, The implementation process of the gated residual fusion mechanism includes: First, the fused features are input into a gating network for processing to generate gating weights. Specifically: ， in, Indicates the number of executions; BN indicates batch normalization; Then, the fused features are processed using the gated graph. Residual fusion is performed with the projected features to obtain the final fused BEV features, specifically: ， in, As a feature of fusion, To integrate BEV features.

8. An electronic device, characterized in that, It includes a processor and a memory, the memory storing program code that, when executed by the processor, causes the processor to perform the steps of the road target detection method of fusion of 4D radar and vision sensors as described in any one of claims 1 to 7.

9. A storage medium, characterized in that, The system contains a computer program or instructions that, when executed on a computer, perform the steps of the road target detection method based on the fusion of 4D radar and visual sensors as described in any one of claims 1 to 7.

Images