Foggy day image sharpening method based on improved MB-TaylorFormer
Through the MB-TaylorFormer network that integrates deep learning and physical imaging models, the performance degradation of foggy image restoration technology in complex haze scenarios is solved, efficient image clarification effect is achieved, and image restoration quality in foggy scenarios of autonomous driving systems is improved.
Patent Information
- Application Number
- CN202510660951.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-21
- Publication Date
- 2025-08-15
AI Technical Summary
The performance of existing foggy-day image restoration technology has dropped sharply in complex haze scenarios, making it difficult to take into account real-time and recovery quality, resulting in an increase in the misjudgment rate of autonomous driving systems in foggy-day scenarios.
Fusion of deep learning and physical imaging models, optimize the MB-TaylorFormer network through multimodal preprocessing, non-local modules and multi-head attention mechanisms, enhance image feature fusion capabilities, and improve image clarity and contrast.
It significantly improves the restoration quality of images in foggy days, with PSNR increased by 13%, and SSIM increased by 11%, supporting high-quality image restoration in a variety of harsh weather conditions.
Smart Images
Figure FT_1 
Figure FT_2 
Figure FT_3
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of computer vision and image processing, and in particular to a foggy image clearing method based on an improved MB-TaylorFormer network. Background Art
[0002] The present invention addresses the key technical bottlenecks in the current field of foggy image restoration: the performance of traditional physical model methods (such as dark channel priors) drops sharply in complex foggy scenes, while mainstream deep learning methods (such as FFA-Net) face the dual challenges of high computational complexity (>100GFLOPs) and insufficient detail preservation. In particular, in practical applications such as autonomous driving and smart cities, existing solutions are difficult to balance the requirements of real-time performance (<30fps) and restoration quality (SSIM>0.8), resulting in a 3-fold increase in the misjudgment rate of systems such as Tesla Autopilot in foggy scenes. These problems are essentially due to technical pain points such as the limitations of atmospheric scattering modeling, the mismatch between the network receptive field and the fog concentration distribution, and insufficient multi-scale feature fusion. The present invention innovatively integrates physical models and deep learning, trains and tests on real foggy traffic scene images to construct an improved MB-TaylorFormer architecture, aiming to break through these key technical bottlenecks that restrict the development of the industry. Summary of the Invention
[0003] To address the challenges of existing technologies, this paper proposes a foggy image sharpening method based on an improved MB-TaylorFormer network. By integrating deep learning with physical imaging models, this method addresses the technical bottlenecks of existing dehazing algorithms, such as detail loss and insufficient contrast in complex atmospheric scattering environments. Through innovative designs such as multimodal preprocessing, attention mechanism optimization, and feature fusion enhancement, the method significantly improves the restoration quality of foggy images.
[0004] The technical solution to achieve the above purpose is:
[0005] Step 1: Construct a multi-source heterogeneous training dataset. Collect real foggy traffic scene images from the Haze-4K dataset and the BeDDE dataset to establish a composite training set.
[0006] Step 2: Multimodal joint preprocessing is used to preprocess the foggy image data. The image enhancement algorithm Retinex, the enhanced atmospheric scattering model (EASM) and the adaptive gamma correction are combined to improve the overall visual effect of the image.
[0007] Step 3: Improve the network architecture by inserting a non-local module at the end of the MB-TaylorFormer encoder to enhance the fusion of global and local features, thereby improving the ability to capture global information. By optimizing the multi-head attention mechanism, the multi-scale feature extraction capability is enhanced to obtain an improved model.
[0008] Step 4: Model training: train the improved model in step 3 with an initial learning rate of 2×10 -4 and gradually decayed to 1×10 through the cosine annealing strategy. -6 , batch size 1, the training network was performed for 160,000 iterations, and all parameters were selected by cross-validation.
[0009] Step 5: Performance Verification. Using the PSNR and SSIM metrics, we compared the improved MB-TaylorFormer with AOD-Net and FFT-Net under the same test conditions. Experimental data showed that the improved MB-TaylorFormer achieved performance improvements: PSNR increased by 13% and SSIM by 11%, validating its superior dehazing performance.
[0010] In step 1, a composite training set is established using real foggy traffic scene images from the Haze-4K dataset and the BeDDE dataset. The data is divided into 70% training set (about 5,600 images), 20% test set (about 1,600 images), and 10% validation set (about 800 images) using a stratified random sampling strategy. The original images are then standardized to a uniform resolution of 512×512.
[0011] In step 2, the multi-scale Retinex (MSR) image enhancement algorithm, the enhanced atmospheric scattering model (EASM), and adaptive gamma correction technology are combined to achieve efficient optimization of foggy images. This method first performs Retinex decomposition based on a Gaussian kernel function to separate the image's illumination and reflection components. MSR then applies filters at different scales to address the image's illumination and detail, specifically using Gaussian filters with varying standard deviations to capture both local and global features. Its formula is:
[0012]
[0013] Where N represents the number of different scales; W n Indicates the weight of each scale; G n (x,y) represents the standard deviation δ of the Gaussian smoothing function at the nth scale n Control the scale.
[0014] The improved EASM model is then used to calculate and optimize the light absorption coefficient; finally, adaptive gamma correction technology is employed. This multimodal fusion preprocessing solution not only significantly improves the visual clarity and contrast of foggy images, but also effectively preserves the detailed features of the original images, providing higher-quality input data for subsequent deep learning processing. This overcomes the limitations of traditional single preprocessing methods in dealing with complex foggy scenes.
[0015] In step 3, the MB-TaylorFormer network architecture was critically improved, mainly including two innovative modules:
[0016] (1) A non-local feature fusion module (Non-Local Block) is innovatively introduced at the end of the encoder. By establishing cross-region feature associations, the interactive fusion capability of global features and local features is effectively enhanced.
[0017] The non-local module is an operation mechanism that can capture global context information. Compared with traditional convolution operations, it goes beyond the limitations of local receptive field. Its basic formula is as follows:
[0018]
[0019] Where x represents the input signal, i.e., the feature map; i represents the index of the input position, i.e., a pixel or a position in the feature map; f(x i ,x j ) represents the similarity measure between positions i and j; g(x j ) represents the representation of the feature map at position j; C(x) represents the normalization factor;
[0020] (2) As the image complexity increases, images of different scales are captured, and the number of attention heads is increased layer by layer to better capture the multi-scale features of spatial details in complex scenes.
[0021] (3) Multi-scale features are fused to generate more refined feature representations. The original attention mechanism is deeply optimized, and a dynamic multi-head attention mechanism with learnable scale factors is proposed to achieve adaptive weighted fusion of multi-scale features.
[0022] These improvements significantly enhance the network's ability to model the global context of images and the efficiency of extracting cross-scale features, enabling the model to achieve better feature expression capabilities while maintaining its original computational efficiency.
[0023] In step 4, the improved model in step 3 is used to train the self-built real foggy dataset. The training is performed on a desktop computer running Ubuntu 18.04, with an Intel(R) Core(TM) i7-11700CPU@2.5GHz, an NVIDIARTX 3090 GPU, and 64G memory. The training hyperparameters are set as follows: the initial learning rate is set to 2×10 -4 , all training samples are resized to 480×480, and the Adam optimizer is used, which is gradually decayed to 1×10 by the cosine annealing strategy. -6 , all parameters are selected through cross-validation, and finally the training weights are obtained.
[0024] In step 5, the improved MB-TaylorFormer was compared with AOD-Net and FFT-Net under the same test conditions using the PSNR and SSIM metrics. Experimental data showed that the improved MB-TaylorFormer achieved performance improvements: PSNR increased by 13% and SSIM by 11%, validating its superior dehazing performance.
[0025] Compared with the existing technology, the present invention has achieved breakthrough progress in the field of image dehazing. Specifically, on the Haze-4K standard test set, the PSNR reaches 30.81dB and the SSIM is as high as 0.987, which are 13% and 11% higher than the current optimal method respectively; at the same time, it supports high-quality image restoration under various severe weather conditions. BRIEF DESCRIPTION OF THE DRAWINGS Figure 1 is a flow chart of the present invention; Figure 2 The location of the Non-Local module of the present invention in the backbone network; Figure 3 It is the structural diagram of T-MSA of the present invention; Figure 4 This is a flow chart of the multi-scale attention optimization (MSAR) module of the present invention; Figure 5 The figure shows the comparison of the defogging evaluation indicators of various methods of the present invention. DETAILED DESCRIPTION
[0026] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments.
[0027] like Figure 1 As shown in the figure, a foggy image clearing method based on the improved MB-TaylorFormer network is implemented as follows:
[0028] Step 1: Construct a multi-source heterogeneous training dataset. Collect real foggy traffic scene images from the Haze-4K dataset and the BeDDE dataset to establish a composite training set.
[0029] Step 2: Multimodal joint preprocessing is used to preprocess the foggy image data. The image enhancement algorithm Retinex, the enhanced atmospheric scattering model (EASM) and the adaptive gamma correction are combined to improve the overall visual effect of the image.
[0030] Step 3: Improve the network architecture by inserting a non-local module at the end of the MB-TaylorFormer encoder to enhance the fusion of global and local features, thereby improving the ability to capture global information. By optimizing the multi-head attention mechanism, the multi-scale feature extraction capability is enhanced to obtain an improved model.
[0031] Step 4: Model training: train the improved model in step 3 with an initial learning rate of 2×10 -4 and gradually decayed to 1×10 through the cosine annealing strategy. -6 , batch size 1, the training network was performed for 160,000 iterations, and all parameters were selected by cross-validation.
[0032] Step 5: Performance Verification. Using the PSNR and SSIM metrics, we compared the improved MB-TaylorFormer with AOD-Net and FFT-Net under the same test conditions. Experimental data showed that the improved MB-TaylorFormer achieved performance improvements: PSNR increased by 13% and SSIM by 11%, validating its superior dehazing performance.
[0033] The specific process of step 1 is as follows:
[0034] The first step is to download the Haze-4K dataset from official sources. This dataset contains 4,000 4K-resolution images of real foggy conditions, covering scenes such as urban roads and highways. Also download the BeDDE dataset, which focuses on driving scenarios in adverse weather conditions, including fog and rain. Select a subset of the dataset related to fog to ensure that it aligns with the target scenes of Haze-4K.
[0035] The second step is to check the integrity of the data, store it by source, and then use perceptual hashing to calculate image similarity, remove duplicate or highly similar images, and divide the dataset into 70% training set, 20% test set, and 10% validation set.
[0036] The specific process of step 2 is as follows:
[0037] The first step is to perform Retinex decomposition based on the Gaussian kernel function to separate the illumination and reflection components of the image. The Multi-Scale Retinex (MSR) image enhancement algorithm addresses the illumination and detail issues of the image by applying filters at different scales. Specifically, Gaussian filters with different standard deviations are used to capture local and global features. The formula is:
[0038]
[0039] Where N represents the number of different scales; W n Indicates the weight of each scale; G n (x,y) represents the standard deviation δ of the Gaussian smoothing function at the nth scale n Control the scale.
[0040] In the second step, the improved EASM model is used to calculate and optimize the light absorption coefficient, i.e., α∈(0,1), to make ASM more robust, thereby improving the defogging effect and correcting the brightness of scenes at different depths. The enhanced ASM formula is as follows:
[0041] I(x,y)=A[1-α(x,y)]·ρ(x,y)·t(x,y)+A[1-t(x,y)]
[0042] Since α decreases with d and its maximum value should be 1, α can be expressed as:
[0043]
[0044] The third step is to use adaptive gamma correction technology to dynamically adjust the brightness distribution according to the global average brightness of the image, thereby enhancing the visual clarity of the image and avoiding local overexposure or underexposure of the image. The gamma correction value γ is calculated as follows:
[0045]
[0046] The specific process of step 3 is as follows:
[0047] In the first step, a non-local feature fusion module (Non-Local Block) is innovatively introduced at the end of the encoder. By establishing cross-region feature associations, the interactive fusion capability of global features and local features is effectively enhanced. Figure 2 shown.
[0048] The non-local module is an operation mechanism that can capture global context information. Compared with traditional convolution operations, it goes beyond the limitations of local receptive field. Its basic formula is as follows:
[0049]
[0050] Where x represents the input signal, i.e., the feature map; i represents the index of the input position, i.e., a pixel or a position in the feature map; f(x i ,x j ) represents the similarity measure between positions i and j; g(x j ) represents the representation of the feature map at position j; C(x) represents the normalization factor;
[0051] In the second step, the number of attention heads is increased layer by layer to better capture the multi-scale features of spatial details in complex scenes, and then the multi-scale features are fused to generate a more refined feature representation. The original attention mechanism is deeply optimized, and a dynamic multi-head attention mechanism with learnable scale factors is proposed to achieve adaptive weighted fusion of multi-scale features. Figure 3 shown.
[0052] The attention weight is calculated by the inner product between the query (Q) and the key (K), and then multiplied by the value (V) to get the final output:
[0053]
[0054] Among them, Q, K, V are feature vectors with size of h×w×D;
[0055] Use a similarity function f(Q i ,K j ) to replace the classic inner product form, Taylor expansion approximation The computational complexity can be reduced from the original O(h 2 w 2 D) is reduced to O(hwD). i ,K j ) Perform a first-order Taylor expansion and we can get the formula:
[0056]
[0057] The computational complexity of the multi-head self-attention (MSA) module and the Taylor expansion-based multi-head self-attention (T-MSA) module when processing h×w image patches is as follows:
[0058] Ω(MSR)=4hwD 2 +2h 2 w 2 D
[0059] Ω(T-MSR)=18hwD+7hwD 2
[0060] Because h×w is usually much larger than D, T-MSA provides more possibilities than MSA when processing high-resolution images and ensures that the results are closer to MSA.
[0061] In the third step, the MSAR module generates a more refined feature representation by fusing multi-scale features. Combined with T-MSA, MSAR further optimizes the attention mechanism and improves the expressiveness of features. Figure 4 As shown:
[0062] The specific process of step 4 is as follows:
[0063] The improved model was used to train a self-built real foggy dataset. The training was performed on a desktop computer running Ubuntu 18.04, with an Intel(R) Core(TM) i7-11700 CPU @ 2.5GHz, an NVIDIA RTX 3090 GPU, and 64G memory. The training hyperparameters were set as follows: the initial learning rate was set to 2×10 -4 , all training samples are resized to 480×480, and the Adam optimizer is used, which is gradually decayed to 1×10 by the cosine annealing strategy. -6 , all parameters are selected through cross-validation, and finally the training weights are obtained.
[0064] The specific process of step 5 is as follows:
[0065] In the first step, the AOD-Net, FFT-Net and original MB-TaylorFormer network models were trained and tested using the same dataset and experimental settings, and the PSNR and SSIM indicators were calculated.
[0066] In the second step, we tested the improved MB-TaylorFormer network architecture. On the Haze-4K standard test set, the PSNR reached 30.81dB and the SSIM was as high as 0.987, which were 13% and 11% higher than the current best method respectively. The indicators are compared as follows: Figure 5 shown.
Claims
1. A foggy image sharpening method based on an improved MB-TaylorFormer, characterized in that: The steps include: Step 1: Construct a multi-source heterogeneous training dataset. Collect real foggy traffic scene images from the Haze-4K dataset and the BeDDE dataset to establish a composite training set. Step 2: Multimodal joint preprocessing is used to preprocess the foggy image data. The image enhancement algorithm Retinex, the enhanced atmospheric scattering model (EASM) and the adaptive gamma correction are combined to improve the overall visual effect of the image. Step 3: Improve the network architecture by inserting a non-local module at the end of the MB-TaylorFormer encoder to enhance the fusion of global and local features, thereby improving the ability to capture global information. By optimizing the multi-head attention mechanism, the multi-scale feature extraction capability is enhanced to obtain an improved model. Step 4: Model training: train the improved model in step 3 with an initial learning rate of 2×10 -4 and gradually decayed to 1×10 through the cosine annealing strategy. -6 , batch size 1, the training network was performed for 160,000 iterations, and all parameters were selected by cross-validation. Step 5: Performance verification: Using PSNR and SSIM indicators, the performance of the improved MB-TaylorFormer is compared with that of the AOD-Net, FFT-Net, and original MB-TaylorFormer network models under the same test conditions.
2. A foggy image sharpening method based on an improved MB-TaylorFormer according to claim 1, characterized in that: In step 1, a composite training set is established using real foggy traffic scene images from the Haze-4K dataset and the BeDDE dataset. The data is divided into 70% training set (about 5,600 images), 20% test set (about 1,600 images), and 10% validation set (about 800 images) using a stratified random sampling strategy. The original images are then standardized to a uniform resolution of 512×512.
3. The foggy image sharpening method based on the improved MB-TaylorFormer according to claim 1, characterized in that: In step 2, efficient optimization of foggy images is achieved by combining the Retinex image enhancement algorithm, the Enhanced Atmospheric Scattering Model (EASM), and adaptive gamma correction technology. This method first performs Retinex decomposition based on a Gaussian kernel function to separate the image's illumination and reflection components. It then uses a modified EASM model to calculate and optimize the light absorption coefficient. Finally, adaptive gamma correction is employed. This multimodal fusion preprocessing scheme not only significantly improves the visual clarity and contrast of foggy images, but also effectively preserves the detailed features of the original image, providing higher-quality input data for subsequent deep learning processing. This overcomes the limitations of traditional single-mode preprocessing methods in handling complex foggy scenes.
4. The foggy image sharpening method based on the improved MB-TaylorFormer according to claim 1, characterized in that: In step 3, the MB-TaylorFormer network architecture was critically improved, mainly including two innovative modules: (1) The non-local feature fusion module (Non-LocalBlock) is innovatively introduced at the end of the encoder. By establishing cross-region feature associations, the interactive fusion capability of global features and local features is effectively enhanced; (2) By increasing the number of attention heads layer by layer, we can better capture the multi-scale features of spatial details in complex scenes. (3) The multi-scale features are fused, the original attention mechanism is deeply optimized, and a dynamic multi-head attention mechanism with learnable scale factors is proposed to achieve adaptive weighted fusion of multi-scale features. These improvements significantly enhance the network's ability to model the global context of images and the efficiency of extracting cross-scale features, enabling the model to achieve better feature expression capabilities while maintaining its original computational efficiency.
5. The foggy image sharpening method based on the improved MB-TaylorFormer according to claim 1, characterized in that: In step 4, the improved model in step 3 is used to train the self-built real foggy dataset. The training is performed on a desktop computer with Ubuntu 18.04, an Intel(R) Core(TM) i7-11700 CPU @ 2.5GHz, an NVIDIA RTX 3090 GPU, and 64G memory. The training hyperparameters are set as follows: the initial learning rate is set to 2×10 -4 , all training samples are resized to 480×480, and the Adam optimizer is used, which is gradually decayed to 1×10 by the cosine annealing strategy. -6 , all parameters are selected through cross-validation, and finally the training weights are obtained.
6. The foggy image sharpening method based on the improved MB-TaylorFormer according to claim 1, characterized in that: In step 5, the improved MB-TaylorFormer was compared with AOD-Net and FFT-Net under the same test conditions using the PSNR and SSIM metrics. Experimental data showed that the improved MB-TaylorFormer achieved performance improvements: PSNR increased by 13% and SSIM by 11%, validating its superior dehazing performance.
7. A foggy image sharpening method based on an improved MB-TaylorFormer according to claim 3, characterized in that The image preprocessing method combines the Multiscale Retinex (MSR) image enhancement algorithm, the Enhanced Atmospheric Scattering Model (EASM), and adaptive gamma correction to improve the overall visual quality of the image. MSR addresses image lighting and detail issues by applying filters at different scales, specifically using Gaussian filters with different standard deviations to capture local and global features. The formula is: Where N represents the number of different scales; W n Indicates the weight of each scale; G n (x,y) represents the standard deviation δ of the Gaussian smoothing function at the nth scale n Control the scale.
8. The foggy image sharpening method based on the improved MB-TaylorFormer according to claim 4, characterized in that: A non-local block is introduced in the decoding stage to enhance the fusion of global features and local features, thereby improving the ability to capture global information. The non-local module is an operation mechanism that can capture global context information. Compared with traditional convolution operations, it goes beyond the limitations of local receptive field. Its basic formula is as follows: Where x represents the input signal, i.e., the feature map; i represents the index of the input position, i.e., a pixel or a position in the feature map; f(x i ,x j ) represents the similarity measure between positions i and j; g(x j ) represents the representation of the feature map at position j; C(x) represents the normalization factor.
9. The foggy image sharpening method based on the improved MB-TaylorFormer according to claim 4, characterized in that: By optimizing the multi-head attention mechanism, the multi-scale feature extraction capability is enhanced. As the complexity of the image increases, images of different scales are captured, and the number of attention heads is increased layer by layer to better capture the multi-scale features of spatial details in complex scenes. The multi-scale features are then fused to generate a more refined feature representation.