An automatic driving image defogging method based on CTFormer

Abstract

The application belongs to the technical field of automatic driving, and particularly relates to an automatic driving image defogging method based on CTFormer.The application comprises the following steps: introducing learnable position coding to pre-process the input fogged image, and enhancing the perception ability of the model to spatial position information; using an improved encoder to model the multi-scale features of the image by combining local convolution and multi-head self-attention mechanism based on Taylor expansion, and capturing global context information and local details; using a decoder to gradually restore high-resolution images by jump connection and multi-scale up-sampling, and combining detail enhancement to strengthen the edges and high-frequency information of the image; and generating clear defogged images through an output module.The application combines the optimization design of contrast learning strategy and multiple loss functions, accurately restores the scene details under foggy conditions, significantly improves the calculation efficiency and robustness of the defogging model, and provides high-quality visual perception input for the automatic driving system.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of automatic driving, and particularly relates to an image defogging method in an automatic driving scene. BACKGROUND

[0002] With the rapid development of automatic driving technology, the visual perception system as one of the key components is crucial for scene understanding, target detection and path planning tasks. However, in adverse weather conditions (such as fog), the image quality collected by the camera is severely degraded, resulting in a significant reduction in the performance of the perception system. This is because fog can cause light scattering and attenuation, making the image exhibit low contrast, blurring and loss of details.

[0003] Traditional image defogging methods can be divided into two categories: physical model-driven and data-driven. For example, the physical model-driven method: such as the atmospheric scattering model, which restores the clear image by calculating the atmospheric transmittance and ambient light component. However, this method is strongly dependent on model parameters, making it difficult to cope with complex real scenes. In addition, data-driven methods based on deep learning have made significant progress in image defogging techniques in recent years. These methods use neural networks to model the global and local features of fogged images, but most of these methods still have the following limitations:

[0004] Insufficient local features: traditional convolutional networks are difficult to capture long-distance context relationships and cannot effectively balance global information and local details in images.

[0005] Weak model generalization ability: relying on specific defogging datasets during the training phase, it is difficult to cope with complex fogging conditions in different scenes.

[0006] The Transformer architecture has achieved success in natural language processing and computer vision due to its excellent global feature modeling capability. In recent years, hybrid models combining convolutional neural networks (CNN) and Transformers have shown great potential in image processing tasks. However, the direct application of existing Transformer models in image defogging tasks still faces the following challenges:

[0007] High computational complexity: traditional multi-head self-attention mechanisms have high computational costs on high-resolution images, making real-time processing difficult.

[0008] Insufficient detail restoration capability: relying only on global feature modeling can easily lead to loss of image detail information. SUMMARY

[0009] The purpose of this invention is to provide an image dehazing method for autonomous driving based on CTFormer, in order to solve the problems of insufficient global information modeling ability, poor detail recovery effect and lack of real-time performance of existing image dehazing technology in autonomous driving scenarios.

[0010] The proposed autonomous driving image dehazing method uses a model combining convolutional feature extraction and an improved Transformer architecture, denoted as CTFormer. Through multi-scale feature modeling, Taylor unfolded attention mechanism (T-MHSA), contrast constraint module (CCM), and detail enhancement module, it achieves efficient and accurate dehazed image generation, significantly improving the quality and processing efficiency of dehazed images, and providing higher robustness and reliability for visual perception in autonomous driving scenarios.

[0011] The CTFormer model includes an input preprocessing module, an encoder module, a decoder module, a contrastive constraint module (CCM), a loss function and optimization module, and an output module; among which:

[0012] Input preprocessing module: such as Figure 5 As shown, the input fogged image is normalized and data augmented. Learnable location encoding is used to enhance the spatial information representation ability of the input features, enabling the model to adapt to different weather and environmental conditions.

[0013] Encoder module: It adopts an improved Transformer architecture, which optimizes the traditional multi-head self-attention mechanism into a high-efficiency attention mechanism based on Taylor unfolding (T-MHSA). It combines local convolution operations to capture multi-scale global features of the image, thereby enhancing the model's ability to represent foggy regions.

[0014] Decoder module: The decoder gradually recovers the high resolution and detail information of the image through multi-scale upsampling and cross-layer skip connections; combined with the detail enhancement module, it further improves the image's clarity and edge performance.

[0015] Contrast Constraint Module (CCM): Utilizes a contrastive learning strategy to construct feature distributions for anchor points, positive samples, and negative samples. By optimizing the relative relationships between features through contrastive loss, it improves the discriminative ability of dehazed features.

[0016] Loss Function and Optimization Module: Design various loss functions, including pixel-level loss, perceptual loss, and contrastive loss, and optimize the model training process through dynamic weight combination to ensure the quality and consistency of the final output image.

[0017] The output module maps the feature map generated by the decoder into a high-definition dehazed image, and combines post-processing techniques such as edge smoothing and detail enhancement to output a high-quality dehazed result that meets the needs of autonomous driving.

[0018] The proposed autonomous driving image dehazing method based on CTFormer includes the following specific steps:

[0019] (1) The preprocessing module performs preprocessing on the foggy image:

[0020] Preprocessing is performed on foggy images acquired in autonomous driving scenarios, transforming the raw images into standardized input tensors that the model can process; specifically including:

[0021] Normalization operation: Normalizes the pixel values ​​of the input image. Transform to the interval [0,1] to eliminate brightness differences between different input images, using the following formula:

[0022] , (1)

[0023] in, and These are the minimum and maximum values ​​of the image pixels, respectively. These are the normalized pixel values.

[0024] Data augmentation: Expand the training dataset in the following ways:

[0025] Random cropping: Cropping a random area of ​​an image while preserving the main information of the image;

[0026] Flip operation: Randomly flips the image horizontally or vertically;

[0027] Add noise: Introduce Gaussian noise or salt-and-pepper noise to simulate sensor noise in a real environment;

[0028] Lighting adjustment: Randomly adjust the image brightness, contrast, and saturation to improve the model's lighting robustness.

[0029] Learnable positional encoding: By incorporating a learnable positional encoding matrix PE, the model can perceive the spatial positional information of the image. The formula for calculating the positional encoding is:

[0030] , (2)

[0031] in, Indicates location, Indicates a dimension index. This is the encoding dimension. The location encoding matrix is ​​added element-wise to the input image tensor, enhancing the model's understanding of spatial relationships.

[0032] (2) encoding the pre-processed image by the encoder module, i.e., inputting the pre-processed image containing fog into the encoder module as an input tensor, and extracting multi-scale features through mixed attention mechanism (such as T-MHSA) and local convolution; the feature map output by the encoder is input into the subsequent module to provide multi-scale and hierarchical feature representation. Specifically:

[0033] extracting multi-scale features of the input image and modeling global dependency by the encoder module; the encoder module combines Taylor-expanded multi-head self-attention mechanism (T-MHSA) and local convolution feature extraction technology, and the specific calculation process includes the following steps:

[0034] (2.1) input feature normalization and blocking operation;

[0035] Layer Normalization: performing layer normalization operation on the input tensor to improve the stability of the features, and the calculation formula is as follows:

[0036] , (3)

[0037] wherein, is the feature mean, is the feature variance, is a stability constant, is the normalized feature tensor.

[0038] Blocking operation: dividing the normalized input image feature into N non-overlapping sub-blocks according to the window size , and the size of each sub-block is .

[0039] , (4)

[0040] (2.2) global feature modeling based on T-MHSA;

[0041] Query, key and value generation: generating query matrix , key matrix and value matrix for each sub-block :

[0042] , (5)

[0043] wherein, is a learnable weight matrix, and d represents the dimension of the query, key and value.

[0044] Attention weight calculation (Taylor expansion optimization): The Softmax function is approximated using Taylor expansion to reduce computational complexity, and the attention score calculation formula is as follows:

[0045] , (6)

[0046] Through Taylor expansion, the Softmax function can be approximated as:

[0047] , (7)

[0048] The calculation formula of attention output is:

[0049] , (8)

[0050] Multi-head attention mechanism: The attention outputs of multiple heads are spliced and linearly transformed, and the calculation formula is:

[0051] , (9)

[0052] where, is the linear transformation matrix, is the number of attention heads.

[0053] (2.3) Hybrid convolution enhanced local feature extraction;

[0054] Local convolution operation: Convolution operation is performed on each sub-block to extract fine-grained features, and the convolution kernel size is 3x3, and the calculation formula is:

[0055] , (10)

[0056] where, is the convolution kernel parameter.

[0057] Local and global feature fusion: The local features extracted by convolution are added to the global features and then processed by layer normalization:

[0058] , (11)

[0059] (2.4) Output of the encoder;

[0060] Multi-scale feature representation: The processed features of the encoder are aggregated through multi-scale convolution to generate feature maps of different resolutions:

[0061] , (12)

[0062] Layer stacking: Multiple encoding layers are stacked to further enhance feature representation capabilities, resulting in a higher output feature map. It will be used as the input to the decoder.

[0063] (3) The decoder module restores the multi-scale features output by the encoder into a high-quality, clear image, and gradually restores the image resolution and details through multi-layer upsampling and detail enhancement modules; the specific process is as follows:

[0064] (3.1) Cross-layer skip connections;

[0065] Feature fusion of skip connections: extracting multi-scale features from each layer of the encoder And pass it to the corresponding layer of the decoder via a skip connection:

[0066] , (13)

[0067] in, This is the output of the layer above the decoder. These are the features after fusion.

[0068] Feature transformation: the transformation of the fused features Transformation using 3×3 convolutions enhances the expressive power of features while reducing redundant information:

[0069] , (14)

[0070] in, These are the convolution kernel parameters.

[0071] (3.2) Multi-scale upsampling

[0072] Upsampling operation: The decoder uses a layer-by-layer upsampling module to restore the low-resolution feature map to a high resolution. Upsampling is achieved using nearest-neighbor interpolation, with the following formula:

[0073] , (15)

[0074] Multi-scale feature fusion: During the upsampling process, a multi-scale fusion module is introduced to combine feature maps of different resolutions, thereby improving the ability to recover details.

[0075] , (16)

[0076] in, Weights for each scale, Indicates the number of scales.

[0077] (3.3) Enhanced details

[0078] High-frequency information extraction: Use the dilated convolution operation to extract high-frequency information to enhance the clarity of details and edges:

[0079] , (17)

[0080] where, is the weight of the dilated convolution.

[0081] Residual enhancement: Combine the original features output by the encoder and the output of the current decoding layer to form a residual connection:

[0082] , (18)

[0083] (3.4) Decoder output

[0084] Restore original resolution: The last layer of the decoder uses a fully connected layer to map the features to the original image size , output clear image:

[0085] , (19)

[0086] Fog removal result output: Limit the image pixel value to the range [0,1] through a nonlinear activation function, generate high-definition image after fog removal.

[0087] (4) Optimize the discrimination ability of features by contrast constraint module (CCM), enhance the comprehensive performance of encoder and decoder through contrast learning strategy, the specific process is:

[0088] (4.1) Anchor sample and positive and negative sample construction;

[0089] Anchor sample definition: Select the global feature of the current input image from the output of the encoder as anchor feature.

[0090] Positive sample definition: Generate enhanced versions of the same input image through random disturbance (such as cropping, rotation, etc.), and extract their features from the corresponding encoder output as positive samples.

[0091] Negative sample definition: Randomly select features from the encoder outputs of other input images as negative samples.

[0092] (4.2) Contrast learning optimization;

[0093] Contrast loss design: The contrast constraint module uses a contrast loss function to optimize the feature distribution between anchor, positive and negative samples, so that the distance between anchor and positive samples is minimized, and the distance with negative samples is maximized. Its core formula is:

[0094] , (20)

[0095] where, is a pre-set margin value, controlling the feature difference between positive and negative samples.

[0096] Feature regularization: During the contrastive learning process, the CCM module adds regularization constraints to the anchor features to avoid overfitting to certain local features by the model:

[0097] , (21)

[0098] Joint loss optimization: Combine the contrastive loss and dehazing loss (such as perceptual loss, pixel loss) with weighted combination to form the total loss function:

[0099] , (22)

[0100] where, and are weight coefficients, controlling the influence of different loss terms.

[0101] (4.3) Feature distance calculation and update;

[0102] Feature distance calculation: In the feature space, use the Euclidean distance to calculate the distance between the anchor and positive samples, negative samples:

[0103] , (23)

[0104] Gradient update: The contrastive constraint module optimizes the model parameters through backpropagation, so that the feature distance meets the contrastive learning goal:

[0105] Objective:

[0106] (4.4) Diversity enhancement of contrastive learning;

[0107] Multi-view positive sample generation: Use data augmentation (such as view change, noise addition, etc.) to generate multiple positive samples, enhancing the model's adaptability to different scenes.

[0108] Dynamic negative sample sampling: Preferentially select samples with closer feature distance to the anchor as negative samples, improving the optimization effect of the contrastive loss.

[0109] (5) Loss function and optimization in the model training process, through the joint design and optimization of multiple loss functions, to improve the quality of the dehazed image and the robustness of the model; the loss function mainly includes pixel-level loss, perceptual loss, contrastive loss and total loss optimization, the specific process is:

[0110] (5.1) Pixel-level loss;

[0111] L1 loss: Directly calculates the absolute difference between the dehazed image and the clear target image at the pixel level, used to maintain pixel consistency.

[0112] Structural Similarity Loss (SSIM): Measures the similarity between the dehazed image and the target image in terms of brightness, contrast and structure, to improve the consistency of visual effects.

[0113] (5.2) Perception loss;

[0114] Feature perception: Use a pre-trained deep convolutional neural network (such as VGG) to extract high-level features of the dehazed image and the target image, and use perception loss to measure the feature difference between the two, emphasizing the semantic consistency of the image.

[0115] Multi-scale perception: Calculate the perception loss from multiple scales to further improve the clarity and detail performance of the dehazed image.

[0116] (5.3) Contrast loss;

[0117] Relative distance between features: Through the contrast learning module (CCM), optimize the feature distance between anchor points, positive samples and negative samples, and enhance the model's ability to distinguish fog images.

[0118] (5.4) Joint optimization of total loss;

[0119] Weighted combination: Weighted combination of pixel-level loss, perception loss and contrast loss to form the total loss function:

[0120] , (24)

[0121] where, , , is the weight coefficient of each loss term, controlling its influence on model optimization.

[0122] Dynamic weight adjustment: Dynamically adjust the weight of the loss term according to the training stage and the output quality of the dehazed image, to ensure the balanced contribution of each loss term in different stages.

[0123] (5.5) Optimization strategy;

[0124] Optimizer selection: Use Adam optimizer combined with momentum term and learning rate decay mechanism to accelerate model convergence.

[0125] Learning rate scheduling: Dynamically adjust the learning rate during training, with a high initial learning rate and a gradual decrease with training iterations, to improve the model's ability to optimize details in the later stage.

[0126] (6) By the output module through mapping and post-processing, the feature map generated by the decoder is converted into the final defogging image to ensure the visual effect and practicality of the output, and the specific process is as follows:

[0127] (6.1) Feature mapping;

[0128] Linear transformation: use a full convolution layer to map the feature map output by the decoder into a clear image consistent with the resolution of the original image:

[0129] , (25)

[0130] where, is the output feature map of the last layer of the decoder, is the weight of the mapping convolution kernel.

[0131] Nonlinear activation: apply an activation function (such as Sigmoid or ReLU) to the output image to limit the pixel value range to [0, 1] to enhance the visual consistency and contrast of the image.

[0132] (6.2) Artifact removal;

[0133] Edge smoothing: smooth the edge area of the generated image to reduce possible artifacts, achieved by a Gaussian filter:

[0134] , (26)

[0135] Detail enhancement: use a Laplacian filter to enhance the high-frequency detail part of the image to optimize the detail performance.

[0136] (6.3) Multi-output level support;

[0137] Low-resolution preview: to meet the real-time requirement, output a low-resolution clear image preview for quick use.

[0138] High-resolution main output: provide a complete high-resolution defogging image for subsequent target detection, path planning, etc. in autonomous driving tasks.

[0139] (6.4) Quality evaluation feedback;

[0140] Image quality evaluation: evaluate the quality of the defogging image based on SSIM (Structural Similarity) and PSNR (Peak Signal-to-Noise Ratio) indicators, and generate an evaluation report to optimize the training of subsequent models.

[0141] Feedback mechanism: automatically adjust model parameters (such as learning rate or loss function weight) based on evaluation results to improve training efficiency.

[0142] The CTFormer-based automatic driving image defogging method of the present application has the following significant advantages:

[0143] (1) Effective combination of global and local features: The encoder module realizes balanced modeling of global context information and local details through the combination of T-MHSA and local convolution, significantly improving the feature expression ability of the fogging area.

[0144] (2) Efficient computation and real-time support: Taylor expansion is used to optimize the attention mechanism, significantly reducing the computational complexity of the model, making it able to meet the real-time processing requirements in the autonomous driving scenario.

[0145] (3) Multi-module collaborative optimization: The combination of the decoder module and the CCM module fully utilizes the contrast learning to enhance the generalization ability of the model, ensuring the consistency of the defogging effect under different scenarios and conditions.

[0146] (4) Image detail recovery and clarity enhancement: The decoder module recovers the image detail information through multi-scale fusion and detail enhancement, generating high-resolution clear defogging images, supporting subsequent target detection and path planning tasks.

[0147] The present application provides an efficient and reliable image defogging solution for autonomous driving systems, with broad application prospects. BRIEF DESCRIPTION OF DRAWINGS

[0148] Figure 1 is the CTFormer model framework diagram of the present application.

[0149] Figure 2 is a multi-scale fusion schematic diagram.

[0150] Figure 3 is a linear Taylor expansion computational complexity schematic diagram of the MSA module.

[0151] Figure 4 and its first-order Taylor expansion curve schematic diagram.

[0152] Figure 5 is a SwinTransformer network architecture schematic diagram. DETAILED DESCRIPTION

[0153] Step one: preprocessing of input image

[0154] First, the fogging images collected by the automatic driving camera system are input into the system. In order to unify the data format and reduce the influence of light changes on the model, the image needs to be normalized, with the pixel value standardized from 0-255 to 0-1. The normalized image is more suitable for subsequent deep learning processing modules. Then, in order to improve the generalization ability of the model, the data set is expanded through data enhancement technology, including random cropping, horizontal flipping, rotation, brightness and contrast adjustment, etc. These enhancement methods can simulate different driving scenarios and camera states, helping the model adapt to complex environments. In addition, adding Gaussian noise and salt and pepper noise can further improve the robustness of the model under camera noise conditions.

[0155] In order to improve the model's perception of spatial information, learnable position encoding is added. Position encoding enhances the representation of spatial features in the image by generating position information for each pixel. After the position encoding is fused with the image features, an input tensor with enhanced spatial perception is formed, providing more complete feature representation for the subsequent encoder module.

[0156] Step two: feature extraction of the encoder module

[0157] The preprocessed image is input into the encoder module of the CTFormer model. The encoder first performs block processing on the image, dividing the input image into fixed-size non-overlapping small blocks. Each small block is sent to a convolutional neural network (CNN) to extract local features such as edges, textures, and shapes. These local features provide the model with detailed information needed for the dehazing task.

[0158] Subsequently, the encoder uses an improved Transformer structure to model the global features of the image. Through the multi-head self-attention mechanism (T-MHSA), the model can capture the context relationship between long-distance regions, thus more comprehensively understanding the image content. T-MHSA optimizes the computational complexity of the attention mechanism through Taylor expansion, enabling the model to efficiently process high-resolution images while maintaining the integrity of feature extraction.

[0159] Finally, the encoder fuses local features with global features to generate hierarchical feature representations through multi-scale convolution operations. These multi-level features contain rich detailed information and global semantic information, providing support for the decoder module.

[0160] Step three: feature optimization of the contrast constraint module

[0161] The features generated by the encoder are further passed to the contrast constraint module (CCM), which optimizes the feature space distribution through a contrastive learning method. First, the features of anchor samples, positive samples, and negative samples are extracted from the encoder output. The anchor sample is the feature representation of the current input image, the positive sample is obtained by slightly enhancing the anchor image (such as brightness adjustment, rotation), and the negative sample is randomly selected from other images.

[0162] The CCM optimizes the feature distance between the anchor sample and the positive and negative samples through a contrastive loss function, making the anchor closer to the positive sample and farther from the negative sample. Through this feature space optimization strategy, the model can more accurately extract the key features of the fogging area and improve the discriminative ability of the dehazing task.

[0163] Step four: feature reconstruction of the decoder module

[0164] The decoder module restores the multi-scale features generated by the encoder to the resolution of the original image through layer-by-layer upsampling and feature fusion. During upsampling, high-resolution features from the encoder are extracted through jump connection and fused with the decoder's features to preserve the image's detail information. The fused features are further processed through convolution operations and activation functions to enhance high-frequency information and edge details.

[0165] The decoder also contains a detail enhancement module that refines the fused feature map. This module enhances the information in the edge and high-frequency regions to generate clearer and more natural dehazing images.

[0166] Step five: dehazing image generation

[0167] The feature map output by the decoder is mapped to a dehazing image, and the pixel value is limited to the range of 0-1 using an activation function such as Sigmoid. To further improve the visual effect of the image, post-processing operations such as edge smoothing, color adjustment, and contrast enhancement are performed on the output image. The final dehazing image not only has high clarity but also retains rich detail information, providing reliable input data for target detection and path planning in autonomous driving scenarios.

[0168] Step six: design and optimization of loss function

[0169] During model training, the design of the loss function is crucial to the dehazing effect. This method uses multiple loss functions, including pixel-level loss, perception loss, and contrast loss. Among them, the pixel-level loss measures the pixel difference between the dehazing image and the clear image to ensure the brightness and color consistency of the output image; the perception loss measures the semantic difference between the two images through the features extracted by the pre-trained model, emphasizing the preservation of image structure and details; and the contrast loss optimizes the feature distribution through contrastive learning, improving the model's discriminative ability for fogging areas.

[0170] The weights of all loss functions are dynamically adjusted according to actual effects, ensuring that the model can focus on optimizing key parts at different training stages. Finally, all loss functions are combined into a total loss function, which optimizes model parameters through backpropagation, gradually improving the dehazing effect.

[0171] Step seven: model testing and deployment

[0172] The trained model will be evaluated on an independent test set to verify its performance in real-world scenarios. The quality of the dehazed images is quantified through indicators such as SSIM (Structural Similarity Index) and PSNR (Peak Signal-to-Noise Ratio), and visual observation is used to ensure the clarity and detail performance of the images.

[0173] Table 1 and Table 2 are the comparison of the model with other methods through experiments.

[0174] As shown in Table 1, in the experimental results on the indoor haze dataset IHAZE, the CT-Former method outperforms other mainstream dehazing methods in various indicators. Specifically, CT-Former achieved scores of 43.02 dB and 0.912 in PSNR and SSIM, respectively, significantly surpassing FFA-Net and TransWeather, demonstrating excellent detail restoration and structure preservation capabilities. At the same time, the NIQE score of CT-Former is 3.47, slightly better than that of USC-Former (3.48), showing good image naturalness. Overall, CT-Former not only achieves high-quality dehazing effects in the IHAZE dataset, but also maintains the naturalness of the images well, verifying its application advantages in indoor complex haze scenarios.

[0175] As shown in Table 2, the experimental results on two outdoor haze data sets (Foggy Haze and Dense Haze) show that the CT-Former method exhibits excellent performance in the dehazing task, and is superior to several existing mainstream dehazing methods, including DCP, AOD-Net, FFA-Net, TransWeather, and USC-Former. Specifically, CT-Former achieves a PSNR of 27.33 dB and an SSIM value of 0.912 in the Foggy Haze dataset, far exceeding other methods, especially in terms of PSNR, CT-Former is 6.08 dB and 3.91 dB higher than FFA-Net and TransWeather, respectively, fully demonstrating its significant advantages in restoring image details and improving image quality. At the same time, the NIQE score of CT-Former is 3.47, close to 3.48 of USC-Former, indicating that while ensuring dehazing effect, CT-Former can also maintain good image naturalness. In the Dense Haze dataset, CT-Former also performs well, with a PSNR of 19.93 dB and an SSIM value of 0.804, again leading all comparison methods, especially in structural similarity. This performance proves that CT-Former can effectively remove haze and restore image details and structure under more difficult haze conditions. Although CT-Former is slightly lower than USC-Former in NIQE index, its score is 3.56, still showing high image naturalness. These results show that CT-Former not only performs well in lighter haze conditions, but also has excellent dehazing ability and image quality preservation ability in more complex Dense Haze environment, demonstrating its potential for wide application in various outdoor haze scenarios.

[0176] The optimized model is exported in a lightweight format (such as ONNX or TensorRT) for deployment on the on-board computing platform of an autonomous vehicle. The model is optimized by pruning and quantization before deployment to ensure that its running speed meets real-time requirements. In actual operation, the system monitors the dehazing effect and performance of the model, and further optimizes the dehazing method by collecting new data and updating model parameters.

[0177] Table 1, results of the method and other methods on indoor dataset IHAZE.

[0178] .

[0179] Table 2, results of the method and other methods on outdoor dataset Foggy Haze and Dense Haze.

[0180] 。

Claims

1. A CTFormer-based automatic driving image defogging method, characterized in that, A model combining convolution feature extraction and an improved Transformer architecture is used, denoted as CTFormer; through multi-scale feature modeling, Taylor expansion attention mechanism (T-MHSA), contrast constraint module (CCM) and detail enhancement module, the defogging image generation is realized; The CTFormer model comprises an input preprocessing module, an encoder module, a decoder module, a contrast constraint module (CCM), a loss function and optimization module and an output module; wherein: The input preprocessing module: normalizes and data enhances the input fogging image, and uses learnable position encoding to enhance the spatial information expression capability of the input feature, so that the model can adapt to different weather and environmental conditions; The encoder module: uses an improved Transformer architecture, optimizes the traditional multi-head self-attention mechanism into an efficient attention mechanism based on Taylor expansion (T-MHSA), and combines local convolution operation to capture multi-scale global features of the image, thereby enhancing the representation ability of the model for fogging areas; The decoder module: gradually restores the high resolution and detail information of the image through multi-scale upsampling and cross-layer skip connection; in combination with the detail enhancement module, the clarity and edge performance of the image are further improved; The contrast constraint module (CCM): uses a contrast learning strategy to construct the feature distribution of anchor points, positive samples and negative samples, optimizes the relative relationship between features through a contrast loss, and improves the discrimination ability of the defogging features; The loss function and optimization module: designs multiple loss functions, including pixel-level loss, perception loss and contrast loss, and combines them with dynamic weights to optimize the model training process, ensuring the quality and consistency of the final output image; The output module maps the feature map generated by the decoder into a high-definition defogging image, and combines post-processing techniques such as edge smoothing and detail enhancement to output high-quality defogging results that meet the needs of autonomous driving; The specific steps are as follows: (1) The pre-processing module pre-processes the foggy image; Specifically, the foggy image collected in the autonomous driving scene is pre-processed to convert the original image into a standardized input tensor that can be processed by the model; (2) The encoder module encodes the pre-processed image, i.e., the pre-processed foggy image is input into the encoder module as an input tensor, and multi-scale features are extracted through hybrid attention mechanism (T-MHSA) and local convolution; the feature map output by the encoder provides multi-scale and hierarchical feature representation for subsequent modules; (3) The decoder module restores the multi-scale features output by the encoder into a high-quality clear image, and gradually restores the image resolution and details through multi-layer upsampling and detail enhancement module; (4) The contrast constraint module (CCM) optimizes the discrimination ability of the features, and enhances the comprehensive performance of the encoder and decoder through a contrast learning strategy; (5) During the model training process, the loss function and optimization are handled, specifically through the joint design and optimization of multiple loss functions, to improve the quality of the defogging image and the robustness of the model; the loss function mainly includes pixel-level loss, perception loss, contrast loss and optimization of total loss. (6) The output module converts the feature map generated by the decoder into the final defogging image through mapping and post-processing to ensure the visual effect and practicability of the output.

2. The automatic driving image defogging method of claim 1, wherein, Step (1) The pre-processing module pre-processes the image containing fog, specifically including: Normalization operation: The pixel values of the input image are converted to the interval [0, 1] to eliminate the brightness difference between different input images, the formula is: Normalization operation: The pixel values of the input image are converted to the interval [0, 1] to eliminate the brightness difference between different input images, the formula is: ， （1） wherein, and Min and Max are the minimum and maximum values of the image pixels, respectively, is the normalized pixel value; Data augmentation: expand the training data set in the following ways: Random cropping: crop a random region of the image, retaining the main information of the image; Flip operation: randomly flip the image horizontally or vertically; Add noise: introduce Gaussian noise or salt and pepper noise to simulate sensor noise in the real environment; Light adjustment: randomly adjust the brightness, contrast and saturation of the image to improve the light robustness of the model; Learnable position encoding: by adding a learnable position encoding matrix PE, the model can perceive the spatial position information of the image, and the calculation formula of the position encoding is: ， （2） wherein, denotes a position, denotes a dimension index, is an encoded dimension; the position encoding matrix is element-wise added to the input image tensor, enhancing the model's understanding of spatial relationships.

3. The automatic driving image defogging method according to claim 2, characterized in that, In step (2), the encoder module extracts multi-scale features of the input image and models global dependency; combined with the multi-head self-attention mechanism based on Taylor expansion (T-MHSA) and local convolution feature extraction technology, the specific calculation process is: (2.1) Input feature normalization and blocking operation; Layer Normalization: on input tensor The layer normalization operation is performed to improve the stability of the features, and the calculation formula is as follows: ， （3） wherein, is a characteristic mean value, is a characteristic variance, is a stability constant, is a normalized feature tensor; Chunking operation: the normalized input image features According to the window size Divided into N non-overlapping sub-blocks The size of each sub-block is ; ， （4） (2.2) Global feature modeling based on T-MHSA; Generation of queries, keys and values: for each sub-block Generation of query matrix Generation of key matrix Generation of value matrix : ， （5） wherein, is a learnable weight matrix, d denotes the dimension of the query, key and value; Attention weight calculation: use Taylor expansion to approximate the Softmax function to reduce the computational complexity, and the attention score calculation formula is as follows: ， （6） Through Taylor expansion, the Softmax function is approximated as: ， （7） The calculation formula of attention output is: ， （8） Multi-head attention mechanism: concatenate and linearly transform the attention outputs of multiple heads, and the calculation formula is: ， （9） wherein, is a linear transformation matrix, is the number of attention heads; (2.3) Hybrid convolution enhanced local feature extraction; Local convolution operation: for each sub-block Convolution operation is performed to extract fine-grained features, and the convolution kernel size is 3x3, and the calculation formula is: ， （10） wherein, are kernel parameters; Local and global feature fusion: the local features extracted by convolution are added to the global features and then normalized by layer normalization ​​ ， （11） (2.4) Output of the encoder; Multi-scale feature representation: the encoder generates feature maps of different resolutions by aggregating the processed features through multi-scale convolution: ， （12） Layer stacking: multiple encoding layers are stacked to further strengthen the feature representation capability, the output feature map will be input as the decoder.

4. The automatic driving image defogging method according to claim 3, characterized in that, The specific process of step (3) is: (3.1) Cross-layer skip connection; Feature fusion of skip connections: extracting multi-scale features from each layer of the encoder And pass it to the corresponding layer of the decoder via a skip connection: ， （13） wherein, is the output of the previous layer on the decoder, is the fused feature; Feature transformation: on the fused features Transformed by 3x3 convolution to enhance the expression ability of features while reducing redundant information: ， （14） wherein, are kernel parameters; (3.2) Multi-scale upsampling Upsampling operation: the decoder uses layer-by-layer upsampling to restore low-resolution feature maps to high-resolution, and uses nearest neighbor interpolation method to realize upsampling, and the formula is: ， （15） Multi-scale feature fusion: in the upsampling process, a multi-scale fusion module is introduced to integrate feature maps of different resolutions to improve the detail restoration capability: ， （16） wherein, is the weight for each scale, denotes the number of scales; (3.3) Detail enhancement High-frequency information extraction: use the thinning convolution operation to extract high-frequency information to enhance the clarity of details and edges: ， （17） wherein, is a weight for the convolution; Residual enhancement: combining original features from the encoder output and the output of the current decoding layer, forming a residual connection: ， （18） (3.4) Decoder output Restore original resolution: The decoder last layer uses a fully connected layer to map the features to the original image size , output clear image: ， （19） Defogging result output: limit the image pixel value to the range of [0, 1] through the nonlinear activation function to generate the high-definition image after defogging.

5. The automatic driving image defogging method according to claim 4, characterized in that, The specific process of step (4) is: (4.1) Anchor sample and positive and negative sample construction; Anchor sample definition: select global feature of current input image from the output of the encoder as anchor feature; Positive sample definition: An augmented version of the same input image is generated by random perturbation, and its features are extracted from the corresponding encoder output as a positive sample; Negative sample definition: randomly select features from the encoder outputs of other input images as negative samples; (4.2) Contrast learning optimization; Contrast loss design: the contrast constraint module uses the contrast loss function to optimize the feature distribution between the anchor, positive and negative samples, so that the distance between the anchor and the positive sample is minimized, and the distance between the anchor and the negative sample is maximized; its formula is: ， （20） wherein, is a preset boundary value, controlling the feature difference between positive and negative samples; Feature regularization: During the contrastive learning process, the CCM module regularizes the anchor features Add regularization constraints to avoid overfitting of the model to certain local features: ， （21） Joint loss optimization: combine the contrast loss and the defogging loss to form the total loss function: ， （22） wherein, and are weight coefficients that control the influence of the different loss terms; (4.3) Feature distance calculation and update; Feature distance calculation: In the feature space, the Euclidean distance between anchor points and positive samples, negative samples is calculated: ， （23） Gradient update: The contrast constraint module optimizes the model parameters through backpropagation, so that the feature distance meets the contrast learning goal: Objectives: ; (4.4) Diversity enhancement of contrast learning; Multi-view positive sample generation: Use data augmentation to generate multiple positive samples to enhance the model's adaptability to different scenarios; Dynamic negative sample sampling: Select samples with close feature distance to anchor points as negative samples to improve the optimization effect of contrast loss.

6. The automatic driving image defogging method according to claim 5, characterized in that, The specific process of step (5) is: (5.1) Pixel-level loss; L1 loss: Directly calculate the absolute difference between the dehazed image and the clear target image at the pixel level to maintain pixel consistency; Structural similarity loss (SSIM): Measure the similarity of brightness, contrast and structure between the dehazed image and the target image to improve the consistency of visual effects; (5.2) Perception loss; Feature perception: Use a pre-trained deep convolutional neural network to extract high-level features of the dehazed image and the target image, and use perception loss to measure the feature difference between the two, emphasizing the semantic consistency of the image; Multi-scale perception: Calculate the perception loss from multiple scales to further improve the clarity and detail performance of the dehazed image; (5.3) Contrast loss; Relative distance between features: Optimize the feature distance between anchor points, positive samples and negative samples through the contrast learning module (CCM) to enhance the model's ability to distinguish foggy images; (5.4) Joint optimization of total loss; Weighted combination: Weighted combination of pixel-level loss, perception loss and contrast loss to form the total loss function: ， （24） wherein, , , are weight coefficients of each loss term, controlling its influence on model optimization; Dynamic weight adjustment: Dynamically adjust the weights of loss terms according to the training stage and the output quality of the dehazed image to ensure balanced contribution of each loss term in different stages; (5.5) Optimization strategy Optimizer selection: Use Adam optimizer combined with momentum term and learning rate decay mechanism to accelerate model convergence; Learning rate scheduling: Dynamically adjust the learning rate during training, with a high initial learning rate and a gradual decrease with training iterations to improve the model's ability to optimize details in the later stage.

7. The automatic driving image defogging method of claim 6, wherein, The specific process of step (6) is: (6.1) Feature mapping; Linear transformation: Use full convolutional layers to map the feature map output by the decoder to a clear image with the same resolution as the original image: ， （25） wherein, is the output feature map of the last layer of the decoder, is the weight of the mapping convolution kernel; Nonlinear activation: Apply an activation function to the output image to limit the pixel value range to [0,1] to enhance the visual consistency and contrast of the image; (6.2) Artifact removal; Edge smoothing: Smooth the edge area of the generated image to reduce possible artifacts, achieved through a Gaussian filter: ， （26） Detail enhancement: Use a Laplacian filter to enhance the high-frequency detail part of the image to optimize detail performance; (6.3) Multi-output level support; Low-resolution preview: To meet real-time requirements, output a low-resolution clear image preview for quick use; High-resolution main output: Provide a complete high-resolution dehazed image for subsequent target detection and path planning in autonomous driving tasks; (6.4) Quality evaluation feedback; Image quality assessment: Based on SSIM and PSNR indicators to evaluate the quality of the dehazing image, and generate an evaluation report to optimize the training of subsequent models; Feedback mechanism: automatically adjust model parameters based on evaluation results to improve training efficiency.

Images