Monocular 3D target detection method for complex weather

Abstract

The invention discloses a monocular 3D target detection method for complex weather, and belongs to the technical field of computer vision and deep learning. According to the method, a weather codebook and a weather adaptive diffusion model are constructed, paired sunny day-foggy day feature training is utilized, and reference features are generated for feature enhancement; and meanwhile, depth position coding and Transform architecture are adopted to fuse the enhanced features and depth information, so that robust target detection in complex weather is realized. In addition, a fog-containing data set is generated based on an atmospheric scattering model for training. According to the method, the accuracy and robustness of monocular 3D target detection under severe weather conditions such as foggy days are effectively improved, reliable target detection under complex weather is realized, and safer visual support is provided for applications such as automatic driving.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision and deep learning, and in particular to a monocular 3D target detection method for complex weather. Background Art

[0002] With the development of autonomous driving technology, the importance of monocular 3D target detection in fields such as autonomous driving and intelligent transportation has become increasingly prominent. Existing monocular 3D target detection methods mainly focus on detection tasks under ideal weather conditions such as sunny days. However, in practical applications, the impact of severe weather conditions (such as foggy and rainy days) on target detection performance cannot be ignored. Under foggy conditions, due to the dense and diffuse characteristics of the fog, light will be strongly scattered and absorbed, resulting in reduced image contrast and visibility, making target detection extremely difficult. In addition, existing methods have deficiencies in feature enhancement and fusion, making it difficult to effectively process visual information under complex weather conditions. Therefore, how to achieve robust and accurate monocular 3D target detection under severe weather conditions has become an urgent problem to be solved. Summary of the Invention

[0003] To address the above problems, the present invention aims to provide a method for robust and accurate monocular 3D target detection under complex weather conditions. By constructing a weather codebook, a weather adaptive diffusion model, and a Transformer feature fusion module, dynamic enhancement and effective fusion of features are achieved, thereby improving the accuracy and robustness of target detection.

[0004] To achieve the above object, the present invention adopts a technical solution: a monocular 3D target detection method for complex weather, comprising the following steps:

[0005] (1) Constructing a weather codebook: Through paired sunny-foggy feature training, the sunny feature matrix and the foggy feature matrix are quantized element by element, generating K embedding vectors of dimension C. The element-by-element quantization is performed based on the nearest neighbor matching of L2 distance to extract the sunny reference features and the foggy reference features.

[0006] (2) Using the weather-adaptive diffusion model to enhance input features: The difference between sunny and foggy features is used as the noise source of fog distribution. Through the forward diffusion and reverse denoising process of T time steps, the cross-attention mechanism is combined to dynamically adjust the feature enhancement strength;

[0007] (3) Depth position code generation: Divide the depth perception feature into D discrete intervals, predict the depth interval probability distribution of each pixel through the classification network, take the embedding vector corresponding to the maximum probability to generate the initial depth code, and superimpose the 3×3 convolution results to obtain the final depth position code;

[0008] (4) Feature fusion and detection: Using the Transformer encoder-decoder architecture, the enhanced features and depth position encoding are input into the encoder and decoder, and the context information and depth features are fused through the multi-head self-attention mechanism and cross-attention module to output the 3D object detection results;

[0009] (5) Dataset generation: Based on the KITTI dataset, different fog concentration parameters are simulated by the atmospheric scattering model to generate a foggy dataset for training the weather adaptive diffusion model and weather codebook.

[0010] The beneficial effects of the above technical solution are as follows: By constructing a weather codebook and a weather-adaptive diffusion model, the present invention can dynamically adjust feature enhancement strategies based on weather conditions, effectively improving the performance and accuracy of monocular 3D object detection in complex weather conditions. Furthermore, the introduction of deep position encoding and the Transformer architecture further enhances the effect of feature fusion, improving the robustness and reliability of detection. Compared with existing technologies, the present invention demonstrates significant performance improvements in adverse weather conditions such as fog, with improved weather adaptability and detection accuracy.

[0011] As a further improvement of the above scheme, the forward diffusion process of the weather adaptive diffusion model is as follows: in the tth time step, the input features and the fog distribution noise are linearly superimposed according to the noise scheduling parameters to generate noisy features. The formula is:

[0012]

[0013] Among them, the initial feature is the sunny feature input, β t is the noise scheduling parameter, Distributes noise for fog.

[0014] The beneficial effect of this technical solution is that, by gradually introducing fog distribution noise through a forward diffusion process, it can simulate feature variations under different weather conditions, providing a more accurate noise model for subsequent feature enhancement. Compared with existing technologies, the forward diffusion process of the present invention can more effectively capture and utilize differences in weather characteristics, thereby achieving more accurate feature enhancement and target detection.

[0015] As a further improvement of the above scheme, the reverse denoising process is as follows: the mean and variance of the fog distribution at the t-th time step are estimated by the denoising autoencoder, with the current features and the weather reference features as input. The formula is:

[0016]

[0017] Among them, the similarity calculation of the cross attention layer adopts the dot product form:

[0018]

[0019] In the formula is the learnable parameter matrix.

[0020] The above technical solution has the beneficial effect of gradually removing noise through the reverse denoising process, effectively restoring and enhancing feature representations, and improving the accuracy and robustness of target detection. Compared with the existing technology, the reverse denoising process of the present invention incorporates a cross-attention mechanism, which can better utilize weather reference features for feature enhancement, thereby achieving more accurate target detection.

[0021] As a further improvement of the above scheme, the total loss function is:

[0022]

[0023] Among them, the recall loss of sunny feature Superposition of sunny feature embedding loss and feature invariance guided loss; adaptive enhancement loss is the mean square error of the fog distribution mean and variance; 3D detection loss Including classification loss, regression loss and depth loss; λ1 and λ2 are balancing hyperparameters.

[0024] The above technical solution has the beneficial effect of comprehensively optimizing model performance by comprehensively considering multiple loss functions, improving the accuracy and robustness of feature enhancement and object detection. Compared with existing technologies, the total loss function of the present invention can better balance the relationship between different tasks, thereby achieving more accurate feature enhancement and object detection.

[0025] As a further improvement of the above scheme, the sunny feature embedding loss generates the probability vector y through global average pooling (GAP) and Softmax operation respectively. c and y r(c) , each element in the vector represents the feature significance probability of the corresponding channel; the sunny feature embedding loss compares the probability vector y based on the KL divergence c and y r(c) The probability distribution of is:

[0026]

[0027] Among them, y c is the probability vector of sunny day features, y r(c) is the probability vector of the sunny reference feature.

[0028] The beneficial effect of the above technical solution is that, through the sunny feature embedding loss, the probability distribution of the sunny feature can be effectively approximated to the probability distribution of the sunny reference feature, thereby improving the learning effect of the weather codebook on the sunny feature. Compared with the existing technology, the sunny feature embedding loss of the present invention can better capture and utilize the saliency information of the sunny feature, thereby achieving more accurate feature enhancement and object detection.

[0029] As a further improvement of the above scheme, the feature invariance guided loss enables the detector to recall features that are irrelevant to the weather but consistent with sunny features through the weather codebook, so as to achieve feature decoupling across weather scenes. The formula is:

[0030]

[0031] Among them, x r(c) is the reference feature of sunny day, x r(f) Reference characteristics for foggy days.

[0032] The beneficial effects of the above technical solution are: through the feature invariance guided loss, it can achieve feature decoupling across weather scenarios, improving the model's generalization and robustness under different weather conditions. Compared with the existing technology, the feature invariance guided loss of the present invention can better capture and utilize feature invariance information, thereby achieving more accurate feature enhancement and object detection.

[0033] As a further improvement to the above scheme, the update mechanism of the weather codebook is as follows: during the training process, by optimizing the sunny feature recall loss and feature invariance guidance loss, the embedding vectors in the weather codebook are dynamically adjusted to improve the accuracy and robustness of feature enhancement.

[0034] The beneficial effect of the above technical solution is that by dynamically adjusting the embedding vectors in the weather codebook, it can better adapt to feature changes under different weather conditions, further improving the effectiveness and robustness of feature enhancement. Compared with the existing technology, the weather codebook update mechanism of the present invention can more effectively capture and utilize knowledge of sunny weather characteristics, thereby achieving more accurate feature enhancement and target detection in complex weather conditions.

[0035] As a further improvement to the above scheme, the method for generating a foggy dataset is as follows: the fog concentration parameter β is set based on the atmospheric scattering model, the light scattering and absorption effects are simulated, and foggy images with different fog concentrations are generated. The formula is:

[0036] I(x)=J(x)t(x)+A(1-t(x))

[0037] Among them, I(x) is the foggy image, J(x) is the clear sky image, t(x) = exp(-βd(x)) is the atmospheric transmittance, d(x) is the depth map, A is the atmospheric light intensity, and β is the fog concentration parameter.

[0038] This formula is derived from the atmospheric scattering model, a physical model used in computer vision and image processing. It effectively describes the propagation characteristics of light in complex weather conditions, such as fog, including its scattering and absorption processes. The formation process of foggy images closely matches this physical model, allowing the formula to accurately simulate the image degradation effects under varying fog concentrations.

[0039] I(x) represents the foggy image, which is the output of the model and reflects the actual light intensity received by the observer under foggy conditions.

[0040] J(x) is a sunny image, which represents the real visual information of the scene under fog-free conditions and is the target information that needs to be restored or detected.

[0041] t(x) is the atmospheric transmittance, which represents the proportion of light that is not scattered or absorbed during its journey from scene objects to the sensor. Its value is affected by factors such as fog density and distance to the object. The greater the distance or the higher the fog density, the lower the transmittance.

[0042] A is the atmospheric light intensity, which reflects the global illumination intensity caused by scattering in the environment and is the main source of the gray fog background in foggy images.

[0043] β is the fog concentration parameter, which directly determines the density of the fog and the degree of light scattering and absorption. By adjusting this parameter, we can generate images with different levels of fog, meeting the model's requirements for data in different weather conditions.

[0044] The beneficial effects produced by the above technical solution are:

[0045] This formula can generate images that are consistent with human visual perception in foggy environments. This enables models trained on this dataset to learn visual features that are closer to actual driving scenarios, thereby improving the model's detection performance in real-world complex weather conditions.

[0046] This method for generating a foggy dataset can simulate foggy scenes with varying fog concentrations, providing rich data support for model training. Compared to existing technologies, the method can more realistically simulate image features under foggy conditions, thereby improving the model's detection performance in foggy scenes.

[0047] As a further improvement of the above scheme, the input of the Transformer encoder-decoder is: the encoder receives the flattened enhanced features, and the decoder receives the features after depth position encoding, and the two types of features are associated through the cross-attention module.

[0048] The beneficial effect of the above technical solution is that, through the Transformer encoder-decoder architecture, enhanced features and deep position encoding can be effectively integrated, improving the effectiveness of feature fusion and the accuracy of object detection. Compared with the existing technology, the Transformer architecture of the present invention can better capture and utilize the global correlation information between features, thereby achieving more accurate object detection.

[0049] As a further improvement of the above scheme, in the inference stage, the parameters of the weather codebook and the weather adaptive diffusion model are fixed, and monocular 3D target detection in complex weather conditions is achieved by dynamically adjusting the feature enhancement strength.

[0050] The above technical solution has the beneficial effect of achieving efficient feature enhancement and target detection during the inference phase by fixing the parameters of the weather codebook and weather-adaptive diffusion model, improving the real-time performance and practicality of the model. Compared with existing technologies, the feature enhancement strategy of the present invention during the inference phase can better adapt to different weather conditions, thereby achieving more accurate target detection.

[0051] Compared with the prior art, the present invention has the following beneficial effects:

[0052] 1. A feature enhancement method based on weather codebook and weather adaptive diffusion model is proposed, which can dynamically adjust the feature enhancement strategy according to weather conditions, effectively improving the performance and accuracy of monocular 3D object detection in complex weather conditions.

[0053] 2. The introduction of deep position encoding and Transformer encoder-decoder architecture further enhances the effect of feature fusion and improves the robustness and reliability of detection.

[0054] 3. A method for generating foggy datasets is proposed, which can simulate foggy scenes with different fog concentrations and provide rich data support for model training.

[0055] 4. In the inference stage, by dynamically adjusting the feature enhancement strength, efficient feature enhancement and target detection are achieved, improving the real-time and practicality of the model.

[0056] In summary, the present invention shows significant performance improvement in adverse weather conditions such as foggy days, has better weather adaptability and detection accuracy, and provides an effective solution for the field of monocular 3D target detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 The overall block diagram mainly includes the weather adaptive diffusion model, weather codebook, and Transformer feature fusion module.

[0058] Figure 2 Schematic diagram of the feature-invariant guidance loss and sunny feature embedding loss proposed during the training process, which enables the weather codebook to effectively reconstruct sunny features.

[0059] Figure 3 Training process of weather adaptive diffusion model: Input sunny day features Add fog variable ∈ n The forward process and the use of weather reference features x r Enhance feature representation quality.

[0060] Figure 4 Encodes the depth position.

[0061] Figure 5 3D detection visualization of this model, MonoDTR, and MonoDETR models in a foggy environment (green: ground-truth annotation box).

[0062] Figure 6 3D and BEV detection visualizations of this model under other complex weather conditions (such as cloudy, dusk, and dust). DETAILED DESCRIPTION

[0063] In order to enable those skilled in the art to better understand the technical solution, the present invention is described in detail below in conjunction with embodiments. The description in this section is only exemplary and explanatory and should not have any limiting effect on the scope of protection of the present invention.

[0064] like Figure 1-6 As shown in FIG, the present invention consists of a weather codebook, a weather adaptive diffusion model and a Transformer feature fusion module.

[0065] First, a weather reference feature is generated from the weather codebook. This feature contains information about the reference weather in the input image. The reference weather serves as a guide for improvement requirements. Sunny days are used as reference weather because they provide clearer and richer visual representations of objects.

[0066] Therefore, this paper proposes a Clear Feature Recalling (CFR) loss function to guide the weather codebook to recall relevant clear weather features and generate weather reference features for any input weather condition (sunny or foggy). Ultimately, the detector can identify the parts of the input features that need improvement based on the weather reference features.

[0067] Secondly, the present invention proposes a weather-adaptive diffusion model that dynamically enhances input feature representation according to weather conditions. Given an input feature (sunny or foggy), the weather-adaptive diffusion model adaptively enhances the input feature representation based on the weather reference feature. Among them, the weather reference feature plays the role of an attention mechanism, guiding the intensity distribution of feature enhancement by quantifying the degree of improvement of the input feature. This design defines the feature difference between sunny and foggy days as fog distribution This is used as a noise source in the diffusion model. By introducing the Adaptive Enhancement (AE) loss, the diffusion model can achieve weather-adaptive feature enhancement through a multi-step inverse process. Ultimately, the model dynamically adjusts the strength of feature enhancement to achieve complex weather detection capabilities.

[0068] Finally, the present invention adopts the Transformer encoder-decoder architecture to globally integrate the enhanced features of the weather adaptive diffusion model and the depth perception features. d The depth position is encoded with the enhanced features and the depth position information of each pixel is embedded into the Transformer to effectively fuse the contextual information and depth-aware features.

[0069] In order to achieve adaptive enhancement of feature representation by utilizing the differences between weather conditions, the present invention generates a foggy KITTI dataset based on the KITTI dataset.

[0070] Figure 1 The overall block diagram of the present invention is shown. The backbone network receives the input image (sunny image or foggy image) and generates the corresponding input feature x (sunny feature x c or foggy weather features x f ). Input this feature into the weather codebook to generate the weather reference feature x r , which is mainly used to guide the degree of enhancement required for the input image features. Subsequently, the weather adaptive diffusion model gradually enhances the input features through T time steps, and finally obtains the enhanced features x. At the same time, the depth perception features x d The enhanced feature x is used to perform depth position encoding. Finally, the Transformer feature fusion module is used to effectively model the relationship between the enhanced feature and the depth perception feature to achieve monocular 3D target detection. c To train the diffusion model and Transformer feature fusion module, the depth perception module uses the depth map to implicitly learn depth perception features during the training phase.

[0071] 1. Weather Codebook

[0072] Under foggy conditions, the overall visual quality of the scene is usually poor, mainly manifested in problems such as reduced contrast and reduced visibility, and significant enhancement is required to restore details. In contrast, under clear weather conditions, due to the rich visual representation information, the expected improvement is relatively small. This difference poses a challenge to the dynamic adaptability of feature enhancement strategies. Therefore, this paper proposes a weather codebook Z, whose core function is to provide a dynamic benchmark for feature enhancement under different weather conditions by learning weather-related reference knowledge. Figure 2 Specifically, the codebook contains K embedding vectors, represented as

[0073]

[0074] Where c represents the dimension of each vector. During the training phase, the model receives paired sunny-foggy features. Sunny feature x c With foggy weather feature x f Generate feature matrix through convolution layer:

[0075] and

[0076] Its spatial dimension is h×w, and the number of channels is c (w represents width, h represents height). Then, for each element of the sunny feature matrix ( and ) is quantized element by element, and the reference features of sunny weather are extracted through the quantization process. Provide a benchmark for subsequent feature enhancement. The calculation formula is as follows:

[0077]

[0078] Furthermore, since the only difference between the clear-day and foggy images is the weather, the quantization process is symmetrical. This process uses the weather codebook to generate foggy reference features with the same semantic structure as the clear-day reference features. To obtain the foggy weather reference features, we perform element-by-element quantization on the foggy features, using the following formula:

[0079]

[0080] Based on sunny feature x c With sunny reference feature x r(c) , this paper proposes the sunny feature embedding (CFE) loss Specifically, the probability vector y is generated by global average pooling (GAP) and Softmax operation respectively. c and y r(c) , each element in the vector represents the feature significance probability of the corresponding channel. c and y r(c) , through KL divergence DKL (·) to compare the probability distribution, the formula is:

[0081]

[0082] Parameter meaning:

[0083] Sunny day feature embedding loss.

[0084] D KL : KL divergence, used to compare the differences between two probability distributions.

[0085] y c : Probability vector of sunny day features.

[0086] y r(c) : Probability vector of sunny reference features.

[0087] This loss minimizes the difference in probability distribution between sunny day features and sunny day reference features, allowing the weather codebook to learn the knowledge of sunny day features and thus reconstruct the corresponding sunny day representation information.

[0088] To ensure that the weather codebook is consistent across weather conditions, this paper proposes an invariant feature guidance loss (IFG) Specifically, this loss enables it to recall features that are not related to the weather but are consistent with sunny features through the weather codebook, thereby achieving feature decoupling across weather scenarios. Its formula can be expressed as:

[0089]

[0090] Finally, superposition and Obtain the sunny feature recall (CFR) loss function The formula can be expressed as:

[0091]

[0092] Parameter meaning:

[0093] Feature invariance guided loss.

[0094] x r(c) : Sunny day reference characteristics.

[0095] x r(f) : Reference characteristics of foggy days.

[0096] The L2 norm squared, which is used to calculate the difference between two features.

[0097] During the training phase, the weight parameters of the K slots embedded in the weather codebook are randomly initialized and updated using formula (5). During the inference phase, all parameters are kept fixed to recover the sunny features, and weather reference features can be generated for any weather condition.

[0098] 2. Weather Adaptive Diffusion Model

[0099] Through the explanation in Section 1, it is clear that the weather codebook can output the reference feature x of sunny day through formula (1) and formula (2) respectively. r(c) Reference characteristics of foggy days x r(f) Since this method can accept input images of any weather conditions, we uniformly represent the weather reference features as x r .

[0100] Figure 3 The core idea of the diffusion model is to gradually transform the noise-containing x into the weather adaptive diffusion model by fixing the T time steps of the Markov chain. T Denoise to target feature x0.

[0101] During the training phase, forward diffusion and reverse denoising processes need to be performed, while during the inference phase, only the reverse process needs to be run.

[0102] This method is different from the traditional diffusion method in which Gaussian noise is applied to the image or latent space. (called fog distribution) as a noise source to enable the model to have foggy perception capabilities. It represents the difference between the same scene under foggy and sunny conditions, so The difference information contained in mainly represents the haze characteristics.

[0103] Through this design, our diffusion model can learn the dynamic changes of weather by adding and removing fog variables. c As the reference input of the diffusion model and given its initial state x0, the initial features are specially relabeled as

[0104] For the forward process at the t-th time step, The previous feature Noise associated with fog (i.e. ) as input to generate This process needs to be iterated for T time steps, and its formula can be expressed as:

[0105]

[0106] Parameter meaning:

[0107] Represents the feature diffusion process from time step t-1 to time step t.

[0108] Represents the fog distribution noise source.

[0109] β t : Noise scheduling parameter, controlling the amount of noise added at each step.

[0110] Features at time step t-1.

[0111] Features at time step t.

[0112] I: Identity matrix.

[0113] Next, in the backward process at the t-th time step, we pass the feature Estimate fog variable ∈ n , thereby enhancing the feature characterization capability of foggy scenes. To this end, the present invention adopts a denoising autoencoder The model receives the features and the reference features x extracted from the weather codebook r . Specifically, ∈ θ By estimating the fog distribution at the tth time step The mean μ θ and variance Σ θ The reverse process will be iterated in T time steps, and its formula can be expressed as:

[0114]

[0115] in The modeling process includes ∈ θ Cross-attention layer in the network.

[0116] Parameter meaning:

[0117] Represents the feature denoising process from time step t to time step t-1.

[0118] Represents the feature distribution after denoising.

[0119] μ θ : Mean estimation function.

[0120] Σ θ : Variance estimation function.

[0121] Features at time step t.

[0122] x r: Weather reference characteristics.

[0123] t: current time step.

[0124] The cross attention layer receives the flattened sunny features. and weather reference feature x r (respectively and ). By calculating and The similarity in the feature space realizes the effective transmission of feature enhancement information, and its formula can be expressed as:

[0125]

[0126] In the formula Where W i q ,W i k ,W i v is the learnable parameter matrix.

[0127] Parameter meaning:

[0128] Q: Query matrix, representing the flattened sunny features.

[0129] K: key matrix, representing weather reference features.

[0130] V: value matrix, representing weather reference features.

[0131] d: feature dimension.

[0132] Softmax: Softmax function, used to calculate attention weights.

[0133] To ensure ∈ θ The estimated fog variable and the imposed fog variable ∈ n Similarly, the present invention proposes an adaptive enhancement loss The formula is:

[0134]

[0135] In the adaptive enhancement loss function Under the constraint of θ The fog distribution is used as the noise source of the diffusion model and multiple forward-backward diffusion processes are used to estimate the fog variables. In addition, ∈ θThe cross-attention layer in the proposed algorithm dynamically enhances feature representation by fusing input features with weather reference features. Because this diffusion model learns the degree of feature improvement required under different weather conditions, it effectively improves the quality of feature representation for any input (whether it is sunny or foggy) during inference. This feature ultimately enables the proposed monocular 3D object detector to achieve robust detection in complex weather conditions.

[0136] 3. Transformer Feature Fusion Module

[0137] This paper constructs a Transformer feature fusion module based on the Transformer encoder-decoder architecture to integrate context information and depth perception features. As introduced in Section 2, we will flatten the enhanced features As input to the Transformer encoder, the encoded context-aware features are obtained through a multi-head self-attention mechanism and calculations using a feed-forward network (FFN). Because depth-aware features contain rich positional information, they can more effectively represent three-dimensional spatial information. Therefore, depth-aware features are used as input to the Transformer decoder. The cross-attention module in the Transformer effectively models the relationship between contextual information and depth features, thereby improving performance.

[0138] The role of depth position encoding in Transformer is to introduce position information and embed the depth position of each pixel into Transformer. Specifically, Figure 4 As shown, the depth perception feature is divided into D discrete intervals, each interval corresponds to a learnable embedding vector The learnable embedding vectors of all intervals form the encoding matrix Each depth interval corresponds to a specific feature embedding space. For each pixel, predict its probability distribution of each depth interval, take the category index with the highest probability, and then find the corresponding code to construct the initial depth position code. To further represent position cues from the local neighborhood, we apply a convolutional layer G with a kernel size of 3×3 to P and add its result to P to finally obtain the deep position encoding.

[0139] 4. Total loss

[0140] The total loss function of the present invention is expressed as:

[0141]

[0142] Parameter meaning:

[0143] Total loss function.

[0144] 3D detection loss.

[0145] Sunny day feature recall loss.

[0146] Adaptive enhancement loss.

[0147] λ1 and λ2: Balancing hyperparameters.

[0148] In our experiments, we set λ1=λ2=1. is the detection loss for 3D object detection. It includes loss functions for classification, regression, and depth loss, similar to previous works. Update the overall weight parameters.

[0149] 5. Experimental Details and Results Analysis

[0150] 5.1. Dataset and Evaluation Method

[0151] Dataset. We used the KITTI 3D object detection dataset, the most widely used dataset for 3D object detection. It contains 7,481 training images and 7,518 test images under clear weather conditions. Due to the lack of ground-truth labels for the test images and the limited evaluation capabilities of the test server, we used 3,712 images as the training set and 3,769 images as the validation set. Furthermore, since our model requires paired images to learn the effects of weather variations, we generated images of simulated foggy scenes from all images in the KITTI dataset.

[0152] In addition, since our work focuses on robust monocular 3D object detection in various weather conditions, we further adopt the Virtual KITTI dataset, which contains realistic synthetic images under various weather conditions (e.g., foggy, cloudy, dusk) and corresponds to the original KITTI dataset.

[0153] Evaluation metrics. We use the average precision (AP) to evaluate 3D detection (AP 3D ) and Bird's Eye View Detection (AP BEV ) performance, and evaluated at three difficulty levels ("easy", "medium", and "hard"), where the difficulty is divided according to the size, occlusion level, and truncation ratio of the target. 40 ), and evaluated on the KITTI dataset with an IoU threshold of 0.7, and on the virtual KITTI dataset with an IoU threshold of 0.5, only detecting the car category.

[0154] 5.2. Experimental Setup and Results

[0155] We use DLA-102 as the backbone network and the depth perception module adopts the DFE module. Adam optimizer (initial learning rate is 10 -4 ) was trained on a single RTX4090 GPU with a batch size of 4 over 120 epochs. For the weather codebook, we used embedding slots K = 4096, with each slot having dimension C = 256. The forward and backward passes of the diffusion model had a time step size of 15, and the diffusion output features had 256 channels.

[0156] The proposed method is compared with the state-of-the-art monocular 3D object detection methods, including GUPNet, DID-M3D, MonoGround, MonoDTR, and MonoDETR, which do not use additional data (such as depth maps or lidar) in the inference phase. We evaluate it on the KITTI validation set in clear sky and the KITTI validation set in foggy weather. Table 1 shows the AP 3D Table 2 shows the results of AP BEV Results. Although recent methods perform well under clear weather conditions, their performance degrades significantly in foggy environments, limiting their applicability in real-world applications. In contrast, our method demonstrates stable detection performance in both foggy and clear weather conditions, validating the robustness of its monocular 3D detection capabilities.

[0157] Table 1: AP of the car category on the KITTI validation set under clear and foggy weather conditions 3D Test results

[0158]

[0159] Table 2: AP of the car category on the KITTI validation set under clear and foggy weather conditions BEV Test results

[0160]

[0161] We also conducted experiments on the virtual KITTI dataset to verify the generalization ability of the proposed method under various weather conditions, such as fog, rain, and dusk. The results, shown in Table 3, show that the proposed model outperforms existing methods in detection under various weather conditions. These experimental results demonstrate that the proposed method exhibits excellent robustness and adaptability under a wide range of weather conditions likely to be encountered in real-world scenarios, not just on sunny days.

[0162] Table 3: Detection results of car categories in the virtual KITTI dataset under foggy, rainy, and dusk weather conditions

[0163]

[0164] We visualized some 3D detection results on the KITTI dataset and compared our model with existing methods MonoDTR and MonoDETR (see Figure 5 Existing methods perform poorly when faced with objects obscured by dense fog, often only detecting fully visible targets, which is a significant limitation. In contrast, the present invention, using a weather codebook and a weather-adaptive diffusion model, can effectively detect fog-obscured targets even in dense fog, demonstrating stronger detection capabilities in complex environments.

[0165] The present invention also provides 3D and BEV detection results under other complex weather conditions (such as cloudy, dusk, and dust) (see Figure 6 The results show that this method can maintain stable detection results under different weather conditions, proving that it also has good weather robustness in real application scenarios and can dynamically enhance the feature representation of the input image.

[0166] It should be noted that, in this article, the terms: include, contain and any other variations are intended to cover non-exclusive inclusion, so that a process, method, article or device that includes a series of elements includes not only those elements, but also other elements that are not explicitly listed, or also includes elements that are inherent to such process, method, article or device. Specific examples are used herein to illustrate the principles and implementation methods of the technical solution of the present invention. The above examples are only used to help understand the method of the present invention and its core idea. The above is only a preferred embodiment of the present invention. It should be pointed out that due to the limitations of textual expression, there are objectively infinite specific structures. For ordinary technicians in this technical field, without departing from the principles of the present invention, several improvements, modifications or changes can be made, and the above technical features can be combined in an appropriate manner; these improvements, modifications, changes or combinations, or the direct application of the concept and technical solution of the present invention to other occasions without improvement, should be regarded as the scope of protection of the present invention.

Claims

1. A monocular 3D target detection method for complex weather conditions, characterized in that: The following steps are involved: (1) Constructing a weather codebook: Through paired sunny-foggy feature training, the sunny feature matrix and the foggy feature matrix are quantized element by element to generate K embedding vectors of dimension C. The element-by-element quantization is based on the nearest neighbor matching of L2 distance to extract sunny reference features and foggy reference features; (2) Using the weather-adaptive diffusion model to enhance input features: The difference between sunny and foggy features is used as the noise source of fog distribution. Through the forward diffusion and reverse denoising process of T time steps, the cross-attention mechanism is combined to dynamically adjust the feature enhancement strength; (3) Depth position code generation: Divide the depth perception feature into D discrete intervals, predict the depth interval probability distribution of each pixel through the classification network, take the embedding vector corresponding to the maximum probability to generate the initial depth code, and superimpose the 3×3 convolution results to obtain the final depth position code; (4) Feature fusion and detection: Using the Transformer encoder-decoder architecture, the enhanced features and depth position encoding are input into the encoder and decoder, and the context information and depth features are fused through the multi-head self-attention mechanism and cross-attention module to output the 3D object detection results; (5) Dataset generation: Based on the KITTI dataset, different fog concentration parameters are simulated by the atmospheric scattering model to generate a foggy dataset for training the weather adaptive diffusion model and weather codebook.

2. The method according to claim 1, characterized in that The specific process of element-by-element quantization in step (1) is as follows: For each element of the sunny feature matrix, calculate its L2 distance with the K embedding vectors in the weather codebook, and select the embedding vector with the smallest distance as the corresponding sunny reference feature; The same operation is performed on the fog feature matrix to generate fog reference features.

3. The method according to claim 1, characterized in that The input of the Transformer encoder-decoder is: The encoder receives the flattened enhanced features, and the decoder receives the features after depth position encoding, and associates the two types of features through the cross attention module.

4. The method according to claim 1, wherein The update mechanism of the weather codebook is as follows: during the training process, the embedding vectors in the weather codebook are dynamically adjusted by optimizing the sunny feature recall loss and feature invariance guidance loss to improve the accuracy and robustness of feature enhancement.

5. The method according to claim 1, wherein During the inference phase, the parameters of the weather codebook and weather adaptive diffusion model are fixed, and monocular 3D target detection in complex weather conditions is achieved by dynamically adjusting the feature enhancement strength.

6. The method according to claim 1, characterized in that The forward diffusion process of the weather adaptive diffusion model is: In the tth time step, the input feature and the fog distribution noise are linearly superimposed according to the noise scheduling parameter to generate the noisy feature, which is formulated as follows: Among them, the initial feature is the sunny feature input, β t is the noise scheduling parameter, Distributes noise for fog.

7. The method according to claim 1, characterized in that The reverse denoising process is: The denoising autoencoder is used to estimate the mean and variance of the fog distribution at the t-th time step. The input is the current feature and the weather reference feature. The formula is: Among them, the similarity calculation of the cross attention layer adopts the dot product form: In the formula is the learnable parameter matrix.

8. The method according to claim 1, characterized in that The total loss function is: in, is the sunny feature recall loss, which is the superposition of the sunny feature embedding loss and the feature invariance guidance loss; is the adaptive enhancement loss, is the mean square error of the fog distribution mean and variance; is the 3D detection loss, including classification loss, regression loss and depth loss; λ1 and λ2 are balancing hyperparameters.

9. The method according to claim 8, characterized in that The sunny feature embedding loss generates the probability vector y through global average pooling (GAP) and Softmax operation respectively. c and y r(c) , each element in the vector represents the feature significance probability of the corresponding channel; the sunny feature embedding loss compares the probability vector y based on the KL divergence c and y r(c) The probability distribution of is: Among them, y c is the probability vector of sunny day features, y r(c) is the probability vector of the sunny reference feature.

10. The method according to claim 8, characterized in that The feature invariance guided loss enables the detector to recall features that are irrelevant to the weather but consistent with sunny features through the weather codebook to achieve feature decoupling across weather scenes. The formula is: Among them, x r(c) is the reference feature of sunny day, x r(f) Reference characteristics for foggy days.

Images