Three-dimensional target detection method in complex weather based on millimeter wave radar and vision fusion

Abstract

Three-dimensional target detection in complex weather is a key problem in intelligent driving, and extreme weather is easy to interfere with a sensor, so that the detection precision is reduced. The invention provides a three-dimensional target detection method in complex weather based on millimeter wave radar and vision fusion, and designs an image generation module for simulating complex environment images such as rainfall, snowfall and the like aiming at the problem of scarcity of bad weather image data in a public data set. And by adopting a self-distillation mechanism, multi-modal knowledge learned by the teacher model in normal weather is migrated to the student model trained in bad weather, so that the detection performance in a complex environment is improved. Besides, in order to enhance the multi-modal information fusion capability, gating fusion modules are introduced into the teacher model and the student model, and a learnable gating fusion mechanism is utilized to dynamically adjust weight distribution of multi-modal feature fusion. According to the method, the robustness and the detection precision of the model under the complex weather condition are effectively improved.

Description

Technical Field

[0001] This invention relates to a three-dimensional target detection method based on millimeter-wave radar and vision fusion under complex weather conditions, which is applicable to target recognition and localization tasks in complex environments in autonomous driving scenarios and intelligent transportation systems. Background Technology

[0002] With the rapid development of autonomous driving technology, 3D target detection has become one of the key tasks for achieving environmental perception. Most existing methods adopt 3D target detection based on a single mode, which relies on a single information source (such as LiDAR or camera). Therefore, under complex weather conditions, the reliability and accuracy of data collected by a single sensor are easily affected by environmental factors such as rain, snow, and fog, resulting in a decrease in detection performance.

[0003] To improve the detection accuracy and robustness of sensing systems, multimodal fusion technology, especially the fusion of radar and cameras, has gradually become a research hotspot. Among these, the fusion methods of millimeter-wave radar and camera (Radar+Camera, abbreviated as C+R) and lidar and camera (LiDAR+Camera, abbreviated as L+C) are the most common. L+C fusion methods typically achieve good 3D target detection performance due to the high precision and high resolution advantages of lidar. However, lidar is costly, and its detection capability is easily affected by significant interference under complex weather conditions such as rain, fog, and snow, impacting its stability and reliability.

[0004] In comparison, although the target detection accuracy of the C+R fusion method is somewhat lower due to the relatively lower spatial resolution and sensing accuracy of millimeter-wave radar, it has good anti-interference capabilities and can maintain stable target detection performance in complex weather environments. To compensate for its accuracy deficiency, recent studies have gradually introduced auxiliary learning mechanisms such as knowledge distillation. By transferring knowledge from high-performance models to C+R-based 3D target detection models, the detection accuracy of C+R-based fusion methods is improved, making its overall performance in complex weather environments approach or even surpass that of L+C fusion methods in some scenarios. Therefore, developing a 3D target detection method based on millimeter-wave radar and camera fusion, designed for complex weather scenarios, and possessing good robustness and detection accuracy has significant practical application value. Summary of the Invention

[0005] This invention proposes a 3D target detection method that integrates millimeter-wave radar and camera for complex weather environments, aiming to improve the robustness and accuracy of 3D target detection under adverse weather conditions. The method mainly includes the following steps:

[0006] A) Acquire spatiotemporally registered millimeter-wave radar data and camera image data as the basic input information for multimodal 3D target detection, and use this to train a teacher model with high detection accuracy to provide cross-modal knowledge supervision;

[0007] B) Considering that the number of publicly available images concentrated under adverse weather conditions such as rain and snow is limited and the actual acquisition cost is high, an image generation module is designed and introduced to synthesize image data under simulated adverse weather conditions (such as rain and snow) based on the normal weather images obtained in step A).

[0008] C) Use the adverse weather images generated in step B) and the normal weather images obtained in step A) together as training samples for image data, while keeping the millimeter-wave radar data obtained in step A) unchanged, so as to train a student model with the same structure as the teacher model, which can adapt to different environmental conditions.

[0009] D) To further improve the expressive power of cross-modal data fusion, a gated fusion network was introduced into both the teacher and student models. This module uses a learnable gating mechanism to dynamically adjust the weight allocation when fusing multimodal information, thereby enhancing the model's ability to suppress inferior modal data and its fusion robustness.

[0010] E) Using a knowledge distillation mechanism, the multimodal knowledge learned by the teacher model trained in step A) under normal weather conditions is transferred to the student model trained in step C) to improve the detection performance of the student model under complex weather conditions.

[0011] F) Save and output the parameter weights of the student model after knowledge distillation in step E) to construct a complex weather-adaptive multimodal 3D target detection model with high robustness and high accuracy.

[0012] In step A), the registered millimeter-wave radar and camera image data can be organized and preprocessed by referring to the annotation format and registration method provided in the public dataset;

[0013] In particular, the teacher model mentioned in step A) and the student model mentioned in step C) both adopt a multi-scale mid-term fusion strategy in terms of multimodal feature fusion. That is, the image and radar data are first encoded by their respective backbone feature extraction networks, and then feature fusion is performed in the mid-level. This avoids feature interference that may be introduced by modal differences in the early fusion, and retains the modal independent learning advantage in the later fusion, thereby improving the robustness of the overall model.

[0014] In step B), the image generation module uses a physical analysis method. Based on the physical properties of natural weather such as raindrops and snowflakes, it constructs corresponding "rain layer" or "snow layer" image masks and merges them with the initial normal weather image at a certain ratio to generate an adverse weather image with rain or snow effects. The specific generation process is shown in equation (1), where I b I represents the final generated image of adverse weather conditions. d The image represents a normal weather image that has been darkened, where L represents the image mask and α represents the fusion ratio. In addition, to improve the diversity and realism of the generated images, multiple random perturbation factors are introduced during the generation process to simulate the changes in rain and snow patterns in the natural environment.

[0015] I b =I d *(1-α)+L*α (12)

[0016] In step C), to better simulate real weather conditions and maintain high detection accuracy of the student model under different environments, after careful experimentation and analysis, a specific ratio of rain, snow and normal weather images in the training samples was designed to make full use of the generated adverse weather image data and enable the student model to maintain good target detection performance under normal weather conditions.

[0017] In step D), the gated fusion module introduces a learnable gated convolutional structure to perform feature transformations on the image and radar features respectively, calculates their respective weight vectors, and calculates the feature values ​​used for subsequent fusion by multiplying them with their respective original features. The specific calculation process is shown in equations (2) and (3), where F C and F R These represent the image and radar features input to the gated fusion module, respectively. σ1 represents the sigmoid function, σ2 represents the ReLU function, and Conv... C and Conv R F represents the convolutional layer corresponding to the two modal features. g c and F g a These represent the generated image and radar gating features, respectively; this module can automatically learn the importance of different modalities in different spatial locations during training, and realize the dynamic adjustment of information fusion weights;

[0018]

[0019] In step E), the knowledge distillation mechanism adopts a self-distillation strategy and designs three distillation loss functions to supervise knowledge transfer. Self-distillation means that students and teachers share a single model, which is different from the traditional complex teacher-student dual-model parallel training method. Specifically, the teacher model is first trained using data under normal weather conditions. After training, all weights of the teacher model are frozen. Then, the multimodal fusion features learned by the teacher model are transferred to the student model trained on a dataset containing images of adverse weather conditions through knowledge distillation. This method effectively reduces the demand for computing resources and shows a significant improvement in the accuracy of 3D target detection of the student model under complex weather conditions in multiple validation scenarios.

[0020] In step F), the model parameters are saved, including all network weights learned during forward inference and backpropagation, and stored as a weight file in a standard loadable format, which facilitates subsequent performance evaluation, model fine-tuning and scientific research reproduction.

[0021] The advantages and / or beneficial effects of the present invention include:

[0022] This invention introduces a simulated adverse weather image generation module, which can synthesize images of complex weather scenarios based on data under normal weather conditions. This solves to some extent the problem of insufficient samples of severe weather such as rain and snow in public datasets, and improves the adaptability and generalization ability of the 3D target detection model in complex environments.

[0023] This invention employs a self-distillation mechanism for cross-modal knowledge transfer, avoiding the high dependence on computational resources in traditional distillation methods. Knowledge transfer can be completed by training only one teacher model, which can significantly reduce training costs.

[0024] This invention introduces a gated fusion module in the feature fusion stage, which enables the model to adaptively adjust the fusion weights according to the importance of different modal features, thereby improving the effectiveness and robustness of radar and image information fusion. Attached Figure Description

[0025] Figure 1 This is a flowchart of a three-dimensional target detection method under complex weather conditions based on millimeter-wave radar and vision fusion, according to an example of the present invention;

[0026] Figures 2-5 This is a comparison chart of experimental results, in which... Figure 2 The results are from an existing 3D target detection model based on millimeter-wave radar and vision fusion under normal weather conditions. Figure 3 , 4 Figures 5 and 6 show the detection results of the student model of this invention under normal, rainy, and snowy weather conditions, respectively. Detailed Implementation

[0027] like Figure 1 As shown, a three-dimensional target detection method based on millimeter-wave radar and vision fusion according to an embodiment of the present invention includes:

[0028] Step 101: Obtain spatiotemporally registered millimeter-wave radar data and camera image data. In a specific embodiment, public datasets such as nuScenes and KITTI can be selected to obtain registered radar and image data, wherein each frame of data contains multiple spatiotemporally synchronized radar point clouds and camera images to construct training sets, validation sets, and test sets with multimodal temporal information;

[0029] Step 102: Based on the dataset constructed in Step 101, refer to open-source frameworks in the field of 3D object detection such as MMDetection3D to establish a standardized data preprocessing process. Convert the data (millimeter-wave radar and camera image data) and its detailed parameters, including image paths, target annotation information and sensor calibration data, into structured format data files to facilitate efficient training and use of multimodal object detection models.

[0030] Step 103: Using the structured data file containing complete data information constructed in Step 102, a teacher model with high detection accuracy based on normal weather conditions is trained as a guiding source for subsequent knowledge transfer. This model adopts a multi-scale mid-term fusion approach of image and radar features. First, features are extracted through their respective backbone networks. Then, the image features are converted from perspective view (PV) to bird's eye view (BEV) with the help of the perspective conversion module. During the perspective conversion process, image depth distribution information and radar supplementary information are combined to improve spatial consistency. Subsequently, a multimodal feature fusion module is established based on a variable cross-attention mechanism to generate feature maps with high spatial resolution and rich semantic information for 3D target detection.

[0031] Step 104: Establish an image generation module and use the camera image data in the structured format data file constructed in Step 102 to simulate image input under adverse weather conditions (such as rain, snow, etc.). Specifically, a physical modeling method is used to simulate weather scenes such as rain and snow. By adjusting the image brightness and saturation and superimposing a "rain layer" or "snow layer" mask with a blurring effect, a highly diverse and realistic synthetic image dataset is constructed. Since millimeter-wave radar is less affected by weather interference, it is not necessary to process the radar data accordingly. According to the adverse weather image generation method in this example, the specific generation process is shown in Equation (4), where Ib represents the final generated adverse weather image, I d This represents a normal weather image that has been darkened, where L represents the image mask and α represents the fusion ratio.

[0032] I b =I d *(1-α)+L*α (15)

[0033] Step 105: Based on the structured format file in step 102 and the data of the synthesized adverse weather images in step 104, train a student model with the same structure as the teacher model. By training the student model under complex weather conditions, improve its adaptability to target detection tasks under interference environments such as rain and snow.

[0034] Step 106: To enhance the adaptive capability of multimodal information fusion, before performing feature fusion operations on the teacher model constructed in Step 103 and the student model constructed in Step 105, a gated fusion network is introduced into both the teacher and student models. First, the image and radar features are aligned according to the channel dimension. Then, they are passed through pre-designed convolutional layers for images and radar respectively, and after passing through the sigmoid activation function, adaptive image and radar weights are obtained. Afterward, the image and radar feature maps that have not passed through the convolutional layers are multiplied by their respective weights to achieve feature weighting. After aligning the image and radar features according to the number of channels, they are passed through convolutional layers containing the ReLU activation function to generate image and radar features (image-gated features and radar-gated features) that can be used for feature fusion. This gated fusion network can dynamically adjust the fusion ratio of each modality in a specific scenario, thereby enhancing the robustness of the model.

[0035] According to one embodiment of the present invention, the method for generating gating features includes the operations shown in equations (5) and (6), wherein F C and F R These represent the image features and radar features input to the gated fusion module, respectively. σ1 represents the sigmoid function, σ2 represents the ReLU function, and Conv... C and Conv R F represents the convolutional layers corresponding to image and radar features, respectively. g c and F g R These represent image-gated features and radar-gated features, respectively. Concat means aligning and merging the image and radar features according to the number of channels.

[0036]

[0037] Step 107: To achieve knowledge transfer under complex weather conditions, a multi-level knowledge distillation strategy based on a self-distillation mechanism is established to guide the training of the student model. This strategy includes three methods: response distillation, mask-based feature scaling distillation, and scene-level distillation. Specifically, response distillation focuses on the prediction results of the target category and the 3D bounding box, and uses Quality Focal Loss (QFL) and Smooth L1 loss to calculate the output difference between the student and teacher models, as shown in equations (7) to (9). Indicates the response distillation loss, Represents classification loss, Representing regression loss, C t and C s B represents the classification result. t and B s The regression result is represented by α, which represents the weight of different target categories in the detection result. Feature scaling distillation based on mask introduces a mask mechanism on top of L2 loss to focus on foreground information and foreground boundary region information, as shown in Equation (10). F represents the feature scaling distillation loss based on the mask. t and F s The feature map is represented by H and W, which represent the height and width of the feature map, and m represents the introduced mask. Scene-level distillation introduces a feature similarity matrix on the basis of L1 loss to capture spatial geometric information and achieve cross-modal knowledge transfer, as shown in equations (11) and (12), where equation (11) is used to calculate the similarity matrix, A i，j f represents the cosine similarity of the similarity matrix at position (i, j). i Let represent the i-th feature vector. Equation (12) uses the similarity matrix obtained from Equation (11) to calculate the loss of feature distillation, where Represents the scene-level distillation loss, A t and A s H and W represent the similarity matrices of the teacher and student models, respectively, and represent the height and width of the feature maps.

[0038]

[0039] Step 108: Save all parameters contained in the student model after knowledge transfer in Step 107, as well as the parameters contained in the distillation module structure designed using the three knowledge distillation methods in the multi-level knowledge distillation strategy, as standard format files (such as .pth or .ckpt) to facilitate convenient model deployment, evaluation, and reproduction. It also facilitates subsequent performance fine-tuning and scientific research expansion. In addition, to simplify the parameter saving process, an automated training framework such as pytorch_lightning can be used for parameter management and saving during training.

[0040] According to a further aspect of the present invention, the image perspective conversion module in step 103 includes sub-modules that perform the following operations respectively:

[0041] Step 1: Process the original image using image depth estimation methods to calculate its depth distribution information D. I , is used to represent the spatial distance distribution of each pixel in an image;

[0042] Step 2: Place D I Image features F under perspective view (PV) I PV Perform the outer product operation to obtain the image features F under the camera's frustum view (FV). I FV ;

[0043] Step 3: Project the radar point cloud onto multi-view images to find corresponding image pixels, then voxelize it into voxels under the camera's cone view. Next, encode the non-empty radar point cloud cylinders into radar feature maps. Finally, perform convolution operations on the radar feature maps and apply a sigmoid activation function to calculate the spatial occupancy of the millimeter-wave radar point cloud. R , used to represent the probability or intensity distribution of point cloud existence in three-dimensional space;

[0044] Step 4: Place O R With F I PV Perform an outer product operation to obtain another set of image features F under the view frustum view. I FV’ Because millimeter-wave radar itself lacks vertical resolution, F... I FV’ The height dimension value is uniformly set to 1 to avoid incorrect space expansion;

[0045] Step 5: To enhance the feature representation ability during the perspective shift process, firstly, for F... I FVPerform a summation operation along the height dimension to compress the height dimension to 1, and then calculate the compressed F. I FV and F I FV’ A concatenation operation is performed along the channel dimension to obtain the camera context (C) that fuses image and radar context information. I FV Calculate C I FV The overall process is shown in equation (13), where This indicates the outer product operation, and Conv represents the convolutional layer;

[0046]

[0047] Step 6: Using the voxel average pooling operation (BEV voxel average pooling) under the bird's-eye view, C I FV The final BEV image features F are obtained by mapping the camera cone view to the bird's eye view (BEV). I BEV This serves as the representation result of the image modality after spatial alignment;

[0048] According to a further aspect of the present invention, the image generation module in step 104 includes sub-modules that respectively perform the following steps:

[0049] Step 1: Transfer the normal weather image I o The grayscale image I is obtained after grayscale transformation. g Extract the brightness distribution and estimate the overall illumination level based on the brightness level of the pixel with the highest proportion in the image;

[0050] Step 2: Convert the image from RGB to HSV color space, reduce the saturation (S channel) and luminance (V channel) according to the lighting level, and then convert it back to RGB to generate a darkened image I. d ;

[0051] Step 3: Based on the illumination levels of the entire image, generate a grayscale image α of the same size as the image, as a weight map representing the "weight coefficient" of each pixel, describing the illumination influence weight of each pixel in the image;

[0052] Step 4: Create a two-dimensional Gaussian noise array, enlarge and crop the initial image I o Apply motion blur and color level adjustment with OTSU threshold to I o To generate a masking layer L simulating raindrops / snowflakes;

[0053] Step 5: Darken image Id By performing the operation of formula (15) with the simulated weather layer L, the image collected under simulated adverse weather conditions is finally obtained;

[0054] According to a further aspect of the present invention, the specific processing steps for merging the structured data and adverse weather image data in step 105 into data used for training the student model include:

[0055] To better simulate real weather conditions and ensure that the student model maintains high detection accuracy under different environments, after careful experimentation and analysis, normal weather images, simulated rainfall images, and simulated snowfall images were combined in a specific ratio. In this example, the ratio is normal weather images: simulated rainfall images: simulated snowfall images = 2:3:5, in order to enhance the robustness and generalization ability of the student model under various environmental conditions.

[0056] According to a further aspect of the invention, the design of the distillation loss function in step 107 includes sub-steps that respectively perform the following operations:

[0057] Step 1: The response distillation loss function includes classification loss and regression loss. The classification loss is for the target category, while the regression loss is for the 3D bounding box. The classification loss is calculated using Quality Focal Loss (QFL), and the regression loss is calculated using Smooth L1 loss.

[0058] Step 2: Feature scaling distillation based on mask. First, a foreground mask is constructed to filter out background interference and improve the efficiency of knowledge transfer. Then, L2 loss is calculated within the mask area to ensure local feature alignment.

[0059] Step 3: Scene-level distillation First, extract the BEV feature maps of teachers and students, then calculate the similarity matrix between their feature vectors, and finally align the overall spatial layout and geometric relationships based on L1 loss to achieve knowledge transfer from the teacher model to the student model.

[0060] The advantages of this invention include:

[0061] (1) By introducing a simulated adverse weather image generation module, complex weather scenes can be effectively synthesized based on data under normal weather conditions, which to some extent solves the problem of insufficient samples of severe weather such as rain and snow in public datasets and improves the adaptability and generalization ability of the three-dimensional target detection model in complex environments.

[0062] (2) The self-distillation mechanism is used for cross-modal knowledge transfer, which avoids the high dependence of traditional distillation methods on computing resources. Only one teacher model needs to be trained to complete the knowledge transfer, which greatly reduces the training cost.

[0063] (3) The present invention introduces a gated fusion module in the feature fusion stage, which enables the model to adaptively adjust the fusion weights according to the importance of different modal features, thereby improving the effectiveness and robustness of radar and image information fusion.

[0064] To verify the effectiveness and improvement of the method of this invention, experiments were conducted. By comparing it with existing 3D target detection models based on the fusion of millimeter-wave radar and vision, the model designed in this invention can achieve more accurate target detection results.

[0065] Figures 2-5 This is a comparison chart of experimental results, in which... Figure 2 The results are from an existing 3D target detection model based on millimeter-wave radar and vision fusion under normal weather conditions. Figure 3 , 4 Figures 5 and 6 show the detection results of the student model of this invention under normal, rainy, and snowy weather conditions, respectively (the green box in the figure represents the ground_truth, and the blue box represents the prediction result of the model). Figure 4 and Figure 5 This demonstrates that the model designed in this invention has a high detection accuracy in rainy and snowy weather. Figure 2 and Figure 3 The comparison shows that the target detection capability of the model designed in this invention is basically equivalent to that of existing models under normal weather conditions.

Claims

1. A method for three-dimensional target detection under complex weather conditions based on millimeter-wave radar and vision fusion, characterized in that... Includes the following steps: Step A) Acquire structured millimeter-wave radar data (R) collected under normal weather conditions, after spatiotemporal registration. o With camera image I o As input information for multimodal 3D object detection, a teacher model with high detection accuracy is trained using a multi-scale mid-term fusion strategy to provide cross-modal knowledge supervision. Step B), using the millimeter-wave radar data R obtained in step A). o With camera image I o Images simulating adverse weather conditions are synthesized using an image generation module based on physical analysis methods. b The generation process is shown in equation (1), I d The image represents a normal weather image that has been darkened. L represents the image mask and α represents the fusion ratio. To improve the diversity and realism of the generated images, several random perturbation factors are introduced during the image generation process, including the image scaling factor that applies motion blur, the image scaling factor that adds Gaussian noise, and the scaling factor that controls the weighted fusion of the simulated weather layer and the darkened image, in order to simulate the changes in rain and snow patterns in the natural environment. I b =I d *(1-a)+L*a (1) in: Camera images under normal weather conditions o The grayscale image I is obtained after grayscale transformation. g Extract grayscale image I g The brightness distribution and according to the grayscale image I g The highest proportion of pixels in the grayscale image are used to estimate the brightness level. g The overall map illumination levels; Camera images under normal weather conditions o Convert from RGB to HSV color space, based on grayscale image I g After reducing the saturation and brightness of the estimated illumination level, the normal weather image I is... o Convert back to RGB to generate a darkened normal weather image I d ; Step C), using the simulated adverse weather image I generated in step B). b And the camera image I obtained in step A) under normal weather conditions o Both are used as training samples for image data, while retaining the millimeter-wave radar data R obtained in step A). o Without changing the model structure, train a student model with the same structure as the teacher model to adapt to complex weather conditions; Step D) In ​​both the teacher model generated in step A) and the student model generated in step C), a gating fusion module is introduced. This gating fusion module dynamically adjusts the fusion weights of multimodal information through a learnable gating mechanism to enhance the model's ability to suppress inferior modal data and its fusion robustness. Step E) Utilize the knowledge distillation mechanism to transfer the multimodal knowledge learned by the teacher model trained in step A) under normal weather conditions to the student model trained in step C) in order to improve the detection performance of the student model under complex weather conditions. Step F) Save and output the parameter weights of the student model after knowledge distillation in step E) to construct a three-dimensional target detection model adapted to complex weather conditions.

2. The method for three-dimensional target detection in complex weather conditions based on millimeter-wave radar and vision fusion according to claim 1, characterized in that, The multi-scale interim fusion strategy employed in step A) includes the following operations: The structured millimeter-wave radar data and camera image data obtained in step A) are organized and preprocessed according to the annotation format and registration method provided in the public dataset. The millimeter-wave radar data and camera image data obtained in step A) are used to extract features through their respective backbone networks; The perspective conversion module converts the image features of the camera image data from perspective view (PV) to bird's-eye view (BEV). During this perspective conversion process, image depth distribution information and radar supplementary information are combined to improve spatial consistency. Based on the variable cross-attention mechanism, cross-modal feature fusion is performed on camera image features and millimeter-wave radar features that have been transferred to the BEV view.

3. The method for three-dimensional target detection in complex weather conditions based on millimeter-wave radar and vision fusion according to claim 1, characterized in that, Step B) further includes: Based on grayscale image I g Estimated illumination levels, generated and normal weather images I o A pixel-level grayscale image α of the same size is used as a weighted image to represent the weight coefficients of each pixel, describing a normal weather image I. o The lighting effect on each pixel influences the weight; Create a two-dimensional Gaussian noise array, enlarge and crop the normal weather image I o Then apply motion blur and color level adjustment with OTSU threshold to I. o To generate a masking layer L simulating raindrops / snowflakes; Darkened normal weather image I d Perform the operation of formula (1) with the mask layer L.

4. The method for three-dimensional target detection in complex weather conditions based on millimeter-wave radar and vision fusion according to claim 1, characterized in that, The specific operation of merging normal weather images and simulated adverse weather images into data for training the student model as described in step C) includes: combining normal weather images, simulated rainfall images, and simulated snowfall images according to a preset ratio to enhance the detection robustness and generalization ability of the student model under various environmental conditions.

5. The method for three-dimensional target detection in complex weather conditions based on millimeter-wave radar and vision fusion according to claim 1, characterized in that, Step D) describes the operation of the gating fusion module in generating multimodal gating features, which includes: Image features F C and radar signature F R Align them according to the channel dimension and concatenate them into a larger feature map; The concatenated feature maps are then processed by convolutional layers (Conv) on pre-designed images. C Convolutional layers for radar R Then, the image weights and radar weights are obtained by passing them through the sigmoid activation function; The initial image features F C and radar signature F R Each feature is weighted by its respective weight. Based on the aligned and weighted image features and radar features according to the number of channels, the image gating feature F is then generated through a convolutional layer containing the ReLU activation function. g C and radar gating feature F g R The specific calculation process is shown in equations (2) and (3), where F C and F R These represent the image features and radar features input to the gated fusion module, respectively. Concat(F) C ,F R ) represents F C and F R Two features are concatenated to generate a larger feature map, where σ1 represents the sigmoid function and σ2 represents the ReLU function. Conv C and Conv R F represents the convolutional layer corresponding to the two modal features. g C and F g R These represent the generated image gating features and radar gating features, respectively.

6. The method for three-dimensional target detection in complex weather conditions based on millimeter-wave radar and vision fusion according to claim 1, characterized in that: The knowledge distillation mechanism described in step E) employs a self-distillation strategy and designs three distillation loss functions to supervise knowledge transfer. The three loss functions are: response distillation, mask-based feature scaling distillation, and scene-level distillation. The self-distillation strategy means that the student model and the teacher model share a single model. Specifically, the teacher model is first trained using millimeter-wave radar data and camera image data under normal weather conditions obtained in step A). ​​Then, all weights of the teacher model are frozen. Finally, the knowledge of the multimodal fusion features learned by the teacher model is transferred to the student model trained on a dataset containing adverse weather images through distillation. The saved parameter weights mentioned in step F) include all network weights learned during forward inference and backpropagation, as well as the weights in the convolutional layers included in the distillation loss function.

7. The method for three-dimensional target detection in complex weather conditions based on millimeter-wave radar and vision fusion according to claim 2, characterized in that, The perspective shift module operations in the multi-scale mid-term fusion strategy include: The depth distribution information D of normal weather images is calculated by processing them using image depth estimation methods. I , is used to represent the spatial distance distribution of each pixel in an image; D I Image features F under perspective view PV I PV Perform the outer product operation to obtain the image features F under the camera cone view FV. I FV ; First, the radar point cloud is projected onto multi-view images to find corresponding image pixels. Then, it is voxelized into voxels under the camera's cone view. Finally, the non-empty radar point cloud cylinders are encoded into radar feature maps F. R PV Finally, by performing a convolution operation on the radar feature map and then connecting it with a sigmoid function, the spatial occupancy information O of the millimeter-wave radar point cloud is calculated. R This is used to represent the probability or intensity distribution of a point cloud in three-dimensional space, and to calculate O. R The overall process is shown in equation (4), where Conv represents the convolutional layer and σ represents the sigmoid function; O R With F I PV Perform an outer product operation to obtain another set of image features F under the view frustum view. I FV’ , will F I FV’ The height dimension value is uniformly set to 1 to avoid incorrect space expansion; To enhance the ability to express features during perspective shifts, we first analyze F... I FV Perform a summation operation along the height dimension to compress the height dimension to 1, and then compress the F... I FV and F I FV’ A concatenation operation is performed along the channel dimension to obtain the view frustum feature C, which fuses image and radar context information. I FV Calculate C I FV The overall process is shown in equation (5), where This indicates the outer product operation, and Conv represents the convolutional layer; Using voxel-average pooling under a bird's-eye view, C I FV The final BEV image features F are obtained by mapping the camera cone view to the bird's-eye view (BEV). I BEV This serves as the representation result of the image modality after spatial alignment.

8. The method for three-dimensional target detection in complex weather conditions based on millimeter-wave radar and vision fusion according to claim 4, characterized in that, The ratio of normal weather images, simulated rainfall images, and simulated snowfall images in the training data for the student model is set as follows: normal weather images: simulated rainfall images: simulated snowfall images = 2:3:

5.

9. The method for three-dimensional target detection in complex weather conditions based on millimeter-wave radar and vision fusion according to claim 8, characterized in that, The operations for calculating distillation loss using the three distillation loss functions include: The response distillation focuses on the prediction results of the target category and the 3D bounding box. The output difference between the student and teacher models is calculated using mass focus loss and smoothing L1 loss, as shown in equations (6) to (8). Indicates the response distillation loss, Represents classification loss, Representing regression loss, C t and C s Representing the classification result, QFL(C) t C s ) represents C t and C s Calculate the mass focus loss QFL, B t and B s Representing the regression results, Representative to B t and B s Calculate the smoothed L1 loss, where α represents the weight of different target categories in the detection results; Feature scaling distillation based on masking introduces a masking mechanism on top of L2 loss to focus foreground information and foreground boundary region information, as shown in Equation (9), where F represents the feature scaling distillation loss based on the mask. t i,j and F s i,j Let H and W represent the feature vectors at position (i,j) in the feature maps of the teacher and student models, respectively. H and W represent the height and width of the feature maps, and m represents the introduced mask. i,j The mask value represents the feature vector at position (i,j); Scene-level distillation introduces a feature similarity matrix on top of L1 loss to capture spatial geometric information and achieve cross-modal knowledge transfer, as shown in equations (10) and (11), where equation (10) is used to calculate the similarity matrix, f i and f j A represents the i-th and j-th eigenvectors respectively. i,j Let represent the cosine similarity of the similarity matrix at position (i,j). Equation (11) uses the similarity matrix obtained from Equation (10) to calculate the loss of feature distillation. Represents the scene-level distillation loss, A t and A s H and W represent the similarity matrices of the teacher and student models, respectively, and represent the height and width of the feature maps.

10. A computer-readable storage medium storing a computer-executable program that enables a processor to perform the method according to any one of claims 1-9.

Images