Weather adaptive three-dimensional target detection method based on multi-modal fusion

Abstract

The invention discloses a weather adaptive three-dimensional target detection method based on multi-modal fusion. The method comprises the following steps: acquiring a multi-modal data set marked with a three-dimensional target true value frame under complex weather; a dual perception fusion detection model is constructed, feature extraction is performed on the laser radar point cloud and the 4D millimeter wave radar point cloud, the features are mapped to a BEV space, a BEV space target distribution heat map is generated through a target perception enhancement module, a 4D millimeter wave radar feature modulation coefficient is generated through a weather perception modulation module, and 4D millimeter wave radar features are modulated. After being spliced with the laser radar features, the laser radar features are input into a decoder, and a target detection result is output; and training the model and carrying out weather adaptive target detection by using the trained model. According to the invention, through dual mechanisms of target space sensing gating and weather condition sensing modulation, the precision of three-dimensional target detection is greatly improved, excellent performance is shown, and the detection robustness is improved in extreme weather such as heavy snow, dense fog, freezing rain and the like.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of three-dimensional target detection, and particularly relates to a weather-adaptive three-dimensional target detection method based on multi-modal fusion. BACKGROUND

[0002] With the rapid rise of the automatic driving industry, three-dimensional target detection has become a key field of research. The main purpose of three-dimensional target detection is to identify and locate objects in three-dimensional space while accurately classifying them. Currently, the two common perception sensors in the automatic driving field, cameras and lidar, will have different degrees of decline in measurement results when facing the interference of external weather such as rain, fog, and snow. 4D millimeter wave radar is suitable for enhancing the robustness of the entire perception system due to its strong anti-interference of measurement results to external weather. However, the measurement accuracy of 4D millimeter wave radar is lower than that of lidar, and the noise is larger. Therefore, it is necessary to balance weather robustness and measurement accuracy in the target detection algorithm to achieve the best overall.

[0003] With the wide application of attention transformation model (Transformer) in the field of deep learning, many works have applied it to the task of target detection. Detection Transformer (hereinafter referred to as DETR) is an end-to-end target detection model proposed by Facebook AI, and its core is based on the Transformer structure. The basic architecture of this method is: a feature extraction backbone module based on convolutional neural network-a feature encoding module based on Transformer-a feature decoding module based on Transformer-a detection head. Among them, the main role of the feature encoding module based on Transformer is to model the global context of image features, enhance the semantic expression of features, and provide more discriminative input for the subsequent decoder. However, since the (class) image is a 2-dimensional dense input, the computational complexity of the core attention operation in this module grows with the square of the input scale, resulting in significantly higher hardware requirements for the algorithm than convolutional neural networks. Therefore, it is necessary to design a method that can integrate global context information and reduce computational complexity to replace this module. SUMMARY

[0004] The present application aims at the deficiencies of the prior art, and proposes a weather-adaptive three-dimensional target detection method based on multi-modal fusion.

[0005] The purpose of the present application is achieved by the following technical solution: a weather-adaptive three-dimensional target detection method based on multi-modal fusion, which comprises the following steps:

[0006] S1, collect a multi-modal data set under complex weather, including laser radar point cloud, 4D millimeter wave radar point cloud and environment image, and label three-dimensional target true value box;

[0007] S2, construct a dual perception fusion detection model, extract features from laser radar point cloud and 4D millimeter wave radar point cloud and map them to BEV space, generate BEV space target distribution heat map through target perception enhancement module to strengthen foreground target features, generate 4D millimeter wave radar feature modulation coefficient through weather perception modulation module and modulate 4D millimeter wave radar features, splice the modulated millimeter wave radar features and laser radar features, and input them into the Transformer decoder to output target detection box, category and confidence;

[0008] S3, train the dual perception fusion detection model using the data set to obtain the trained dual perception fusion detection model;

[0009] S4, use the trained dual perception fusion detection model to realize weather adaptive target detection.

[0010] Further, the feature extraction from laser radar point cloud and 4D millimeter wave radar point cloud and mapping to BEV space includes: voxelizing the input laser radar and 4D millimeter wave radar point cloud, then inputting the voxels of the two modalities into independent three-dimensional sparse convolution layers, batch normalization and activation functions respectively;

[0011] Then, the features of the two modalities are input into the same backbone network composed of sparse convolution network blocks, and the output three-dimensional sparse features of each network block are projected to BEV space through the BEV mapping block to obtain 4D millimeter wave radar BEV features and laser radar BEV features

[0012] Further, the BEV space target distribution heat map generated by the target perception enhancement module to strengthen the foreground target features specifically includes:

[0013] The BEV features of each modality or respectively use a convolution layer Conv 1×1 with a convolution kernel of 1x1 to reduce the dimension of the feature channels with a group normalization layer GN to obtain low-dimensional features or The formula is:

[0014]

[0015] The low-dimensional features of the two modalities (where i ∈ {L, R}, representing LiDAR and 4D mmWave radar, respectively, hereinafter), two branches are used, the first branch is a heat map predictor MaskPredictor composed of multiple layers of perception, and the second branch is a 1 × 1 convolution kernel gated convolution layer Conv 1×1 , each of which predicts a heat map mask1, mask2; the value range of the two heat maps is limited to 0 to 1, and then the heat map mask1 is added to the learnable bias parameter β value range of 0 to 1 and element-wise multiplied to obtain the gating weight W enc , which is element-wise multiplied with the low-dimensional feature to obtain the enhanced feature The formula is:

[0016]

[0017] Further, the 4D mmWave radar feature modulation coefficient is generated by the weather perception modulation module, and the 4D mmWave radar feature is modulated, specifically including:

[0018] For the input image Image, a lightweight two-dimensional convolutional neural network classifier CLS is used to classify the current weather, and a weather probability distribution Logit is output, the formula is:

[0019] Logit = CLS(Image)

[0020] , which is element-wise multiplied with the low-dimensional feature list to obtain the enhanced feature weather ; during training, the output of the weather classifier CLS is supervised according to the true value label label through the cross-entropy loss function CrossEntropyLoss(, ), the formula is:

[0021] loss img = CrossEntropyLoss(CLS(Image), label).

[0022] Further, the feature modulation coefficient is multiplied with the 4D mmWave radar point cloud feature, and the LiDAR point cloud is spliced, including: the feature modulation coefficient Score weather is multiplied with the 4D mmWave radar enhanced feature , and then connected with the LiDAR enhanced feature to obtain the integrated BEV feature F BEV , the formula is:

[0023]

[0024] where Concat is the connection function.

[0025] Further, the input to the Transformer decoder obtains the bounding box, its class and confidence, specifically including:

[0026] For the enhanced features, use the detection head to predict 1000 bounding boxes, their classes and confidences as the initial query, and the enhanced features as the key and value, input to the Transformer decoder, which is stacked by 6 identical decoder layers. The features output by each layer are predicted by the detection head to obtain new bounding boxes as the query of the next layer. The high-dimensional features obtained by the Transformer decoder are input to the detection head to predict the final bounding boxes, their classes and confidences.

[0027] Further, when training the target perception enhancement module, use the BEV ground truth box to generate a Gaussian heat map label mask gt to supervise the heat map predictor MaskPredictor branch, guide the heat map mask1 output by the heat map predictor to be similar to mask gt ; the loss function loss score used for supervision consists of two parts:

[0028] The first part is the weighted binary cross-entropy loss loss BCE , which is used to increase the loss weight of positive samples to alleviate the class imbalance problem. The weight W l is α times the value of mask gt , and the formula is:

[0029] W l =1+α·mask gt

[0030]

[0031] The second part is the KL divergence loss loss KL , which is used to constrain the overall consistency between the predicted distribution and the target distribution. Normalize mask1 and mask gt to and respectively, and then calculate the KL divergence loss loss KL , and the formula is:

[0032]

[0033]

[0034] The calculation formula of the loss function loss score of the heat map predictor MaskPredictor is:

[0035] loss score =loss BCE +loss KL .

[0036] Furthermore, the E Gaussian heat map label mask generated by the BEV truth box gt Specifically:

[0037] For the true value label GT 3D =[x,y,z,l,w,h,θ], where [x,y,z] is the center coordinate of the label box, [l,w,h] is the length, width and height of the label box, and θ is the deflection angle of the label box. Compress it to the BEV grid space, with the grid length and width dx and dy respectively, to get GT BEV ; The formula is:

[0038]

[0039] For each grid point (i, j) in the BEV space, calculate its distance to GT BEV The rotation offset of the center [Δx rot Δy rot ] T ; The formula is:

[0040]

[0041] According to GT BEV The size of the box and the preset scaling factor σ scale , calculate the variance σ of the corresponding Gaussian kernel x , σ y , and calculate the relationship between each grid point (i, j) and the GT BEV The Gaussian response value of the box is gauss(i,j); the formula is:

[0042] σ x =l′*σ scale ,σ y =w′*σ scale

[0043]

[0044] For each grid point (i, j), take all valid GT BEV The maximum Gaussian value of the box and the Sigmoid function are applied to get the mask gt ; The formula is:

[0045]

[0046] In another aspect, the present application also provides a weather adaptive three-dimensional target detection device based on multi-modal fusion, comprising a memory and one or more processors, the memory stores executable code, and the processor executes the executable code to realize the weather adaptive three-dimensional target detection method based on multi-modal fusion.

[0047] In another aspect, the present application also provides a computer readable storage medium having a program stored thereon, which, when executed by a processor, realizes the weather adaptive three-dimensional target detection method based on multi-modal fusion.

[0048] Advantages of the present application:

[0049] The image-controlled laser radar-4D millimeter wave radar adaptive feature adjustment module provided by the present application realizes a good balance between the measurement accuracy of the laser radar and the weather robustness of the 4D millimeter wave radar by dynamically adjusting the relative proportion of the features of the two modal sensors, thereby improving the adaptability of the three-dimensional target detection method to complex weather outside. The real label guided gating module provided by the present application replaces the complex Transformer encoding module in the original DETR type target detection method with a simple gating module, and introduces real labels into the training process to guide the model to better focus on the features of the target area, thereby reducing the computational complexity while improving the performance of the model. Under the same data and training parameter settings, the present application achieves the most advanced performance on the K-Radar dataset. BRIEF DESCRIPTION OF DRAWINGS

[0050] Figure 1 The network overall structure diagram of the weather adaptive three-dimensional target detection method based on multi-modal fusion provided by the present application is shown in the figure.

[0051] Figure 2 The three-dimensional voxel-based sparse convolutional neural network backbone module provided by the present application is used to extract laser radar and 4D millimeter wave radar features.

[0052] Figure 3 The target perception enhancement module structure diagram provided by the present application is shown in the figure.

[0053] Figure 4 The weather perception modulation module structure diagram provided by the present application is shown in the figure.

[0054] Figure 5 The weather adaptive three-dimensional target detection device based on multi-modal fusion provided by the present application is shown in the figure. DETAILED DESCRIPTION

[0055] In order to make the above objectives, characteristics and advantages of the present application more obvious and easy to understand, the specific embodiments of the present application are described in detail below with reference to the drawings.

[0056] In the following description, a large number of specific details are set forth in order to facilitate a thorough understanding of the present application, but the present application can also be implemented in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the concept of the present application, therefore the present application is not limited to the specific embodiments disclosed below.

[0057] The present application proposes a weather adaptive three-dimensional target detection method based on multi-modal fusion, Figure 1 is a schematic diagram of the overall structure of the method, and the network is suitable for three-dimensional target detection tasks of laser radar-4D millimeter wave radar fusion, especially automatic driving scenes in outdoor bad weather.

[0058] The method specifically comprises:

[0059] S1, collecting a multi-modal data set under complex weather, containing laser radar point cloud, 4D millimeter wave radar point cloud and environment image, and labeling three-dimensional target true value box;

[0060] S2, constructing a dual perception fusion detection model, extracting features from laser radar point cloud and 4D millimeter wave radar point cloud and mapping them to BEV space, generating BEV space target distribution heat map through a target perception enhancement module to strengthen foreground target features, generating 4D millimeter wave radar feature modulation coefficient through a weather perception modulation module and modulating 4D millimeter wave radar features, splicing the modulated millimeter wave radar features and laser radar features, and inputting them into a DETR decoder to output target detection box, class and confidence;

[0061] S3, training the dual perception fusion detection model using the data set to obtain the trained dual perception fusion detection model;

[0062] S4, using the trained dual perception fusion detection model to realize weather adaptive target detection.

[0063] The dual perception fusion detection model comprises a three-dimensional voxel-based sparse convolutional neural network backbone module, a target perception enhancement module, a weather perception modulation module, and a Transformer decoder and detection head.

[0064] The three-dimensional voxel-based sparse convolutional neural network backbone module extracts features of the input laser radar and 4D millimeter wave radar point cloud, and maps the features from three-dimensional space to BEV space;

[0065] The target perception enhancement module generates a BEV space target distribution heat map through a heat map predictor, generates a Gaussian heat map label to supervise the training of the heat map predictor, and multiplies the heat map with a heat map obtained through another convolution-Sigmoid branch element by element to obtain a gating weight and multiply the gating weight with the BEV feature to strengthen the foreground target feature.

[0066] The weather perception modulation module uses a light convolutional neural network classifier to perform weather recognition on the environment image, outputs a weather probability distribution, weights the probability distribution with a preset weather weight parameter tensor, generates a 4D millimeter wave radar feature modulation coefficient, and splices the 4D millimeter wave radar feature after modulation with the laser radar feature to realize dynamic fusion.

[0067] The Transformer decoder refers to SEED, uses a detection head to predict 1000 detection boxes and their categories and confidence as initial queries, and inputs the enhanced features as keys and values into the Transformer decoder (which is stacked by 6 identical decoder layers, and the output features of each layer are predicted by the detection head to obtain new detection boxes as queries of the next layer) to obtain high-dimensional features. The high-dimensional features are input into the detection head to predict the final detection boxes and their categories and confidence.

[0068] The three-dimensional voxel-based sparse convolutional neural network backbone module specifically comprises:

[0069] (1) The input laser radar and 4D millimeter wave radar point clouds are voxelized, and then the voxels of the two modalities are input into independent input network blocks, which include three-dimensional sparse convolutional layers, batch normalization and activation functions;

[0070] (2) Then, the features of the two modalities are input into the same backbone network composed of a plurality of sparse convolutional network blocks, and the three-dimensional sparse features output by each network block are projected into the BEV space through a BEV mapping block to obtain a plurality of groups of BEV features of the two modalities. Then, all BEV features of each modality are spliced to obtain 4D millimeter wave radar BEV features and laser radar BEV features Wherein, the backbone network structure refers to the backbone network structure of Voxel R-CNN, and the specific structure and parameters are as shown in Figure 2 and Tables 1 and 2.

[0071] Table 1: Sparse Convolutional Neural Network Backbone Module Parameters

[0072]

[0073]

[0074] Table 2 BEV mapping module parameters

[0075]

[0076] The target perception enhancement module structure is as shown in Figure 3 , and specifically comprises:

[0077] The BEV features of each modality or are respectively processed by a convolution layer Conv with a convolution kernel of 1x1 1×1 and a group normalization layer GN to reduce the dimension of the feature channels, to obtain low-dimensional features or The formula is:

[0078]

[0079] The low-dimensional features of the two modalities (where i∈{L,R}, respectively representing lidar and 4D millimeter wave radar, as hereinafter), are respectively processed by two branches, i.e. ① a heat map predictor MaskPredictor composed of a multilayer perception (MLP), and ② a gated convolution layer Conv with a convolution kernel of 1x1 1×1 and a Sigmoid function, each predicting a heat map mask1, mask2. The value range of the two heat maps is limited between 0 and 1, and then the heat map mask1 is added with a learnable bias parameter β (value range 0 to 1) and multiplied element by element to obtain a gating weight W enc , which is multiplied element by element with the low-dimensional features to obtain enhanced features The formula is:

[0080]

[0081] The weather perception modulation module structure is as shown in Figure 4 , and specifically comprises:

[0082] For the input image Image, a lightweight two-dimensional convolutional neural network classifier CLS is used to classify the current weather, and a weather probability distribution Logit is output, the formula being:

[0083] Logit = CLS(Image)

[0084] which is weighted and summed with a preset weather weight parameter tensor Score list to generate a 4D millimeter wave radar feature modulation coefficient Score weather .

[0085] The feature modulation coefficient is multiplied by the 4D millimeter wave radar point cloud feature, and then spliced with the laser radar point cloud, specifically:

[0086] The feature modulation coefficient Score weather is multiplied by the 4D millimeter wave radar enhanced feature , and then connected with the laser radar enhanced feature to obtain the integrated BEV feature F BEV , the formula is:

[0087]

[0088] In the training process, in addition to the supervision loss of the output features of each layer of the Transformer decoder consistent with the baseline model SEED, the supervision loss of the detection head, etc., the dual perception fusion detection model also specially designs supervision losses for the target perception enhancement module and the weather perception modulation module, specifically:

[0089] The weather perception modulation module: the output of the weather classifier CLS is supervised by the cross-entropy loss function according to the true value label label, the formula is:

[0090] loss img = CrossEntropyLoss(CLS(Image), label)

[0091] The target perception enhancement module: use the BEV true value box to generate a Gaussian heat map label mask gt to supervise the heat map predictor MaskPredictor branch, guide the heat map mask1 output by the heat map predictor to be as similar as possible to mask gt . The loss function loss score used in supervision is composed of two parts: ① the weighted binary cross-entropy loss loss BCE , which is used to enhance the loss weight of the positive sample (the non-zero area of mask gt ), and alleviate the class imbalance problem, the increase value of the weight W l is α times the value of mask gt , the formula is:

[0092] W l = 1 + alpha * mask gt

[0093]

[0094] ② KL divergence loss loss KL , which is used to constrain the overall consistency of the prediction distribution and the target distribution, and normalize mask1 and mask gt respectively With Subsequently, the KL divergence loss loss KL is calculated, and the formula is as follows:

[0095]

[0096] The loss function loss of the heat map predictor MaskPredictor score is calculated as follows:

[0097] loss score = loss BCE + loss KL .

[0098] Wherein, the BEV heat map mask generated by the real label is gt , and the specific formula is as follows:

[0099] For each data frame, each real label GT of the "small car" category to be detected 3D = [x, y, z, l, w, h, θ], wherein [x, y, z] is the center coordinate of the label box, [l, w, h] is the length, width and height of the label box, and θ is the deflection angle of the label box. Compress it to the BEV grid space (the grid length and width are dx and dy, respectively), to obtain GT BEV ; the formula is as follows:

[0100]

[0101] For each grid point (i, j) in the BEV space, the rotational offset [Δx rot Δy rot ] T of the center of the GT BEV to the grid point is calculated; the formula is as follows:

[0102]

[0103] According to the size of the GT BEV box and the preset scaling factor σ scale , the variances σ x , σ y of the corresponding Gaussian kernel are calculated, and the Gaussian response value gauss(i, j) of each grid point (i, j) to the GT BEV box is calculated; the formula is as follows:

[0104] σ x = l' * σ scale , σ y = w' * σ scale

[0105]

[0106] For each grid point (i,j), take all valid GT BEV The maximum Gaussian value of the box, and apply the Sigmoid function to get the mask gt The formula is:

[0107]

[0108] Compared with other similar three-dimensional target detection methods based on the Transformer, the parameter amount and the calculation amount of the target perception enhancement module are shown in Table 3.

[0109] Table 3 Comparison of parameter amount and calculation amount of Transformer encoder in similar methods

[0110]

[0111] The K-Radar dataset is an autonomous driving dataset for three-dimensional target detection and tracking in complex weather, containing 58 scenes and about 35,000 frames of data. Available sensors include four stereo cameras, two laser radars and one 4D millimeter wave radar, etc. The dataset has seven weather types of sunny, overcast, rainy, foggy, freezing rain, light snow and heavy snow, with day and night data. For three-dimensional target detection, the dataset supports five categories of small cars, buses or trucks, motorcycles, bicycles and pedestrians. The following settings are used for the present application and the comparative methods mentioned below: in the front and rear directions 0-72 meters, both sides directions-6.4-6.4 meters, height direction-2-6 meters, the voxel is divided into a grid size of 0.4 meters, and the small car category target is detected. The evaluation index of the algorithm is the AP index, and under the confidence threshold of 0.3, the AP index of the three-dimensional and BEV of the algorithm result is evaluated with the IoU of 0.3 and 0.5 respectively.

[0112] In order to reflect the progressiveness of the method proposed in the present application, the results of the baseline SEED model and other most advanced models are compared on the K-Radar dataset, and the results are shown in Table 4:

[0113] Table 4 Comparison of results of the method of the present application and other methods

[0114]

[0115] From Table 4, it can be seen that the present application has obvious improvement in accuracy compared with other most advanced three-dimensional target detection methods for complex weather scenes.

[0116] Corresponding to the foregoing embodiment of the weather adaptive three-dimensional target detection method based on multi-modal fusion, the present application also provides an embodiment of a weather adaptive three-dimensional target detection device based on multi-modal fusion.

[0117] Referring to Figure 5 The embodiment of the present application provides a weather adaptive three-dimensional target detection device based on multi-modal fusion, including a memory and one or more processors, the memory stores executable codes, and the processor executes the executable codes to implement the weather adaptive three-dimensional target detection method based on multi-modal fusion in the foregoing embodiment.

[0118] The embodiment of the weather adaptive three-dimensional target detection device based on multi-modal fusion provided by the present application can be applied to any device with data processing capability, which can be a device or apparatus such as a computer. The device embodiment can be realized by software, or by hardware or a combination of software and hardware. Taking software realization as an example, as a logical device, it is formed by reading the corresponding computer program instructions in the non-volatile memory into the memory for execution by the processor of the device with data processing capability where it is located. From the hardware level, as shown in Figure 5 As shown in the figure, in addition to the processor, the memory, the network interface, and the non-volatile memory shown in the figure, the device with data processing capability where the embodiment is located usually includes other hardware according to the actual functions of the device with data processing capability, and details are not described here. Figure 5 As shown in the figure, in addition to the processor, the memory, the network interface, and the non-volatile memory shown in the figure, the device with data processing capability where the embodiment is located usually includes other hardware according to the actual functions of the device with data processing capability, and details are not described here.

[0119] The implementation process of the functions and roles of each unit in the above device is specifically described in the implementation process of the corresponding steps in the above method, and details are not described here.

[0120] For the device embodiment, since it basically corresponds to the method embodiment, the related part can be referred to the part of the method embodiment. The device embodiment described above is only schematic, and the units described as separate components can or can not be physically separated, and the components displayed as units can or can not be physical units, that is, they can be located in one place, or distributed on multiple network units. According to actual needs, part or all of the modules can be selected to achieve the purpose of the present application scheme. Those skilled in the art can understand and implement without creative labor.

[0121] The embodiment of the present application also provides a computer readable storage medium, which stores a program, and the program is executed by a processor to realize the weather adaptive three-dimensional target detection method based on multi-modal fusion.

[0122] The computer readable storage medium can be an internal storage unit of any data processing capable device, such as a hard disk or a memory, of any of the preceding embodiments. The computer readable storage medium can also be an external storage device of any data processing capable device, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. Further, the computer readable storage medium can include both an internal storage unit and an external storage device of any data processing capable device. The computer readable storage medium is used to store the computer program and other programs and data required by the data processing capable device, and can also be used to temporarily store data that has been output or will be output.

[0123] The present application also provides a computer program product, which comprises a computer program, and the computer program is executed by a processor to realize the weather adaptive three-dimensional target detection method based on multi-modal fusion.

[0124] Other embodiments of the present application will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. The specification and examples given are exemplary and are not intended to limit the scope of the application. The true scope of the application is indicated by the claims.

[0125] It should be understood that the general description above and the following detailed description are only exemplary and explanatory, and are not limiting to the present application. The present application is not limited to the precise structures described and shown in the drawings, and various modifications and changes can be made without departing from the scope thereof. The scope of the present application is only limited by the appended claims.

Claims

1. A weather-adaptive three-dimensional target detection method based on multimodal fusion, characterized in that: The method comprises the following steps: S1. Collect multimodal datasets under complex weather conditions, including lidar point clouds, 4D millimeter-wave radar point clouds, and environmental images, and annotate 3D target ground truth boxes. S2. Build a dual-perception fusion detection model to extract features from the lidar point cloud and 4D millimeter-wave radar point cloud and map them to the BEV space. Generate a BEV space target distribution heat map through the target perception enhancement module to enhance foreground target features. Generate 4D millimeter-wave radar feature modulation coefficients and modulate the 4D millimeter-wave radar features through the weather perception modulation module. Concatenate the modulated millimeter-wave radar features with the lidar features and input them into the Transformer decoder to obtain the detection box, its category, and its confidence. S3. Using the data set to train the dual-perception fusion detection model to obtain a trained dual-perception fusion detection model; S4. Use the trained dual-perception fusion detection model to achieve weather-adaptive target detection.

2. The weather-adaptive three-dimensional target detection method based on multimodal fusion according to claim 1, characterized in that: The feature extraction of the lidar point cloud and the 4D millimeter-wave radar point cloud and mapping them to the BEV space includes: voxelizing the input lidar and 4D millimeter-wave radar point clouds, and then inputting the voxels of the two modalities into independent three-dimensional sparse convolution layers, batch normalization, and activation functions respectively; The features of the two modalities are then input into the same backbone network composed of sparse convolutional network blocks, and the output 3D sparse features of each network block are projected into the BEV space through the BEV mapping block to obtain the 4D millimeter wave radar BEV features. BEV features with LiDAR 3. The weather-adaptive three-dimensional target detection method based on multimodal fusion according to claim 1, characterized in that: The generating of the BEV spatial target distribution heat map by the target perception enhancement module specifically includes: BEV characteristics for each mode or Use the convolution layer Conv with a convolution kernel of 1×1 respectively 1×1 The feature channel is reduced in dimension by the group normalization layer GN to obtain low-dimensional features. or The formula is: Low-dimensional features of the two modalities Where i∈{L,R}, represents the laser radar and 4D millimeter wave radar respectively, the same below), using two branches respectively. The first branch is the heat map predictor MaskPredictor composed of a multi-layer perceptron, and the second branch is the gated convolution layer Conv with a convolution kernel of 1×1. 1×1 The Sigmoid function predicts a heat map mask1 and mask2 respectively; the value range of the two heat maps is limited to between 0 and 1, and then the heat map mask1 is added with a learnable bias parameter β with a value range of 0 to 1 and multiplied element by element to obtain the gate weight W enc , and combine it with the low-dimensional features Element-wise multiplication to obtain enhanced features The formula is: W enc =(mask1+β)⊙mask2) 4. The weather-adaptive three-dimensional target detection method based on multimodal fusion according to claim 1, characterized in that: Generating a 4D millimeter-wave radar feature modulation coefficient and modulating the 4D millimeter-wave radar feature by the weather perception modulation module specifically includes: For the input image Image, a lightweight two-dimensional convolutional neural network classifier CLS is used to classify the current weather and output the weather probability distribution Logit. The formula is: Logit=CLS(Image) Compare it with the preset weather weight parameter tensor Score list Weighted summation to generate the 4D millimeter wave radar characteristic modulation coefficient Score weather , the feature modulation coefficient is multiplied by the 4D millimeter wave radar point cloud feature and then spliced ​​with the lidar point cloud; during training, the output of the weather classifier CLS is supervised by the cross entropy loss function CrossEntropyLoss(,) according to the true value label label, and the formula is: loss img =CrossEntropyLoss(CLS(Image),label)。 5. The weather-adaptive three-dimensional target detection method based on multimodal fusion according to claim 4 is characterized in that: The characteristic modulation coefficient is multiplied by the 4D millimeter wave radar point cloud feature and then stitched with the laser radar point cloud, including: weather Enhanced features with 4D millimeter wave radar After multiplication, the laser radar enhanced features Connect them to get the integrated BEV feature F BEV , the formula is: Concat is the concatenation function.

6. The weather-adaptive three-dimensional target detection method based on multimodal fusion according to claim 1, characterized in that: The input to the Transformer decoder obtains the detection box and its category and confidence, specifically including: For the enhanced features, the detection head is used to predict the detection box, its category, and confidence as the initial query. The enhanced features are used as keys and values ​​and input into the Transformer decoder. The Transformer decoder is composed of 6 identical decoder layers stacked together. The features output by each layer are predicted by the detection head to predict a new detection box as the query for the next layer. The high-dimensional features finally obtained by the Transformer decoder are input into the detection head to predict the final detection box, its category, and confidence.

7. The weather-adaptive three-dimensional target detection method based on multimodal fusion according to claim 3, characterized in that: When training the target perception enhancement module, the BEV truth box is used to generate a Gaussian heat map label mask. gt To supervise the heat map predictor MaskPredictor branch, guide the heat map mask1 and mask output by the heat map predictor gt Similar; loss function used in supervision score It consists of two parts: Part 1: Weighted Binary Cross Entropy Loss BCE , used to enhance the loss weight of positive samples and alleviate the problem of category imbalance. The weight W l The added value is mask gt α times the value of , the formula is: W l =1+α·mask gt Part 2: KL divergence loss KL , used to constrain the overall consistency of the predicted distribution and the target distribution, respectively mask1 and mask gt Normalized to and Then calculate the KL divergence loss KL , the formula is: Loss function of heat map predictor MaskPredictor loss score The calculation formula is: loss score =loss BCE +loss KL 。 8. The weather-adaptive three-dimensional target detection method based on multimodal fusion according to claim 7, characterized in that: The E Gaussian heatmap label mask generated by the BEV ground truth box gt Specifically: For the true value label GT 3D =[x,y,z,l,w,h,θ], where [x,y,z] is the center coordinate of the label box, [l,w,h] is the length, width and height of the label box, and θ is the deflection angle of the label box. Compress it to the BEV grid space, with the grid length and width dx and dy respectively, to get GT BEV ; The formula is: For each grid point (i, j) in the BEV space, calculate its distance to GT BEV The rotation offset of the center [Δx rot Δy rot ] T ; The formula is: According to GT BEV The size of the box and the preset scaling factor σ scale , calculate the variance σ of the corresponding Gaussian kernel x , σ y , and calculate the relationship between each grid point (i, j) and the GT BEV The Gaussian response value of the box is gauss(i,j); the formula is: s x =l′*σ scale ,s y =w′*σ scale For each grid point (i, j), take all valid GT BEV The maximum Gaussian value of the box and the Sigmoid function are applied to get the mask gt ; The formula is: Where k is the kth label of the current frame, and valid is the valid GT BEV The total number of boxes.

9. A weather-adaptive three-dimensional target detection device based on multimodal fusion, comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that: When the processor executes the executable code, it implements a weather-adaptive three-dimensional target detection method based on multimodal fusion according to any one of claims 1 to 8.

10. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, a weather-adaptive three-dimensional target detection method based on multimodal fusion according to any one of claims 1 to 8 is implemented.

Images