All-in-one method for removing severe weather by fusing two-stage converter with double-time-phase characteristics

Abstract

The invention discloses an all-in-one method for removing severe weather by fusing a two-stage converter with a double-time-phase feature, and relates to the technical field of artificial intelligence image processing and computer vision. Compared with a traditional image restoration method, the method does not need to manually provide priori knowledge, and enables the model to have powerful performance and more accurate restoration results in the restoration task of the weather degradation image by means of the powerful learning capability of the neural network; compared with other deep learning image restoration methods, the method has the advantages that reasonable design and expansion are performed on the basis of Transformer, the proposed FDFormer network can utilize channel and space two-stage attention to enable the model to capture feature information more comprehensively and enhance details, robustness of the model to different weather and different data sets is enhanced, and the method is suitable for being applied to deep learning image restoration. Compared with other excellent weather degradation image restoration methods, the method has better performance and generalization ability; and the method also has good performance for challenges such as unlearned severe environments and the like.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence image processing and computer vision technology, and specifically to an all-in-one method for removing bad weather by fusing dual-phase features with a two-stage converter. Background Art

[0002] The all-in-one bad weather removal problem is a field that has emerged in recent years. Since computer vision tasks are often affected by rain, fog, snow and other weather conditions, complex degradation will occur under these weather conditions, including low light, blur, noise, low contrast and other problems that reduce vision, making it challenging to capture high-quality images. In many scenarios, such as military image capture, rescue in rugged terrain, etc., it is obvious that low-quality image effects cannot meet the needs of actual scenarios. In order to deal with different types of degradation, people have proposed different methods to solve the bad weather removal of images.

[0003] For example, in image deraining, some use conditional generative adversarial networks (GANs) for image deraining, some use Gaussian processes to transfer learning from synthetic rainfall data to real-world rainfall data, and some use complementary cascade networks to remove rain streaks and raindrops in a unified framework. In weather defogging, an enhanced Pix2pix network based on GAN is proposed to generate fog-free images, and contrast regularization learning technology and dynamic feature enhancement modules are used to remove haze. For snow removal, Desnow-Net is one of the earliest proposed CNN-based image desnowing methods. On this basis, a stacked dense network for snow removal has emerged, and JSTASR has introduced a size and transparency perception method for snow removal. However, the actual situation is often complex and changeable, usually involving two or more types of weather conditions and other complex environmental factors. For example, rain is often accompanied by fog, haze, and other weather conditions, resulting in blurring and artifacts in the captured images. Although these methods show good performance under specific single conditions, they are only applicable to certain typical weather conditions. Single-task image processing methods cannot meet the needs of the real world, especially tasks such as military image acquisition. Therefore, image restoration under simultaneous multi-task processing has become an important focus in various computer vision applications.

[0004] Specifically, some use a generator consisting of multiple task-specific encoders and a shared encoder to restore the image. Some use a separate encoder-decoder structure to restore images affected by various atmospheric conditions. This type of method will lead to a rapid increase in model size in practical applications, because each weather requires its own encoder. So some propose to automatically select a pipeline for weather-degraded data to enhance the existing model, and use two different stages to learn general weather and specific weather. There are also methods for designing effective image restoration network architectures, such as Multi-StageProgressive and MAXIM, which are general image restoration methods, and they have also successfully solved a series of severe weather conditions. However, these schemes limit the restoration performance under specific weather conditions, and the performance is significantly reduced in actual scenarios. Although the Transformer-based method is more efficient than the task-specific encoder network, its performance is limited by the lack of fusion and utilization of features at different scales of the network, and the locality of spatial information may be missing, which will also affect the image restoration results.

[0005] In summary, in the field of applied technology, image restoration tasks still have some limitations:

[0006] (1) Most Transformer-based image weather removal methods are limited to self-attention. The self-attention operation itself is used for natural language processing. In image restoration, it is necessary to convert the two-dimensional image data into sequence data. The high dimensionality and locality of image data often lead to overfitting problems when the model is processed. In addition, the amount of calculation is large and limited to a fixed spatial range. Although it promotes the fusion of contextual information of different spatial scales, it makes concessions in effective memory utilization.

[0007] (2) In order to solve the limitations of the model, some use position encoding to process the spatial relationship of the image, which solves the limitations of the model to a certain extent, but lacks the use of global information. Therefore, they use self-attention along the channel dimension or within a fixed spatial range block to reduce the computational load. However, this trade-off leads to limitations in capturing long-range spatial features, resulting in poor image restoration results.

[0008] (3) Currently, most severe weather environment datasets are mainly manually selected and randomly extracted from single image task datasets, which contain a limited variety of complex weather conditions, which are different from real environments. Therefore, it becomes crucial to collect real severe weather datasets. However, the availability of public datasets representing real severe environments is limited, which leads to unsatisfactory training results for many models. In addition, existing datasets usually cannot cover all weather conditions. In actual missions such as military and rescue operations, various scenarios may coexist, which is also a challenge.

[0009] Therefore, a new solution to the above problems needs to be proposed. Summary of the invention

[0010] The object of the present invention is to provide an all-in-one method for removing bad weather by fusing dual-phase characteristics with a two-stage converter to solve the technical problems raised in the background technology.

[0011] To achieve the above object, the present invention provides the following technical solution: an all-in-one method for removing bad weather by integrating dual-phase characteristics with a two-stage converter, comprising at least the following steps:

[0012] S1: Batch preprocessing of degraded images in the dataset;

[0013] S2: Merge the original dataset with the synthetic dataset to perform non-overlapping dataset division;

[0014] S3: Design a network architecture for feature extraction and image restoration;

[0015] S4: Design a loss function for reconstructing images to improve image similarity;

[0016] S5: Use the trained model to perform measurement verification, and the model training is completed;

[0017] S6: Restoration is completed by inputting the image to be degraded into the model.

[0018] Preferably, the batch preprocessing of the degraded images in the data set in S1 at least comprises the following steps:

[0019] Randomly crop the original degraded image, and correspond the corresponding labels to the randomly cropped images to ensure that the labels and coordinate cropped images have the same size;

[0020] The images and labels are enhanced, and then converted into tensors through the conversion channel. The processed images and labels are used for model training.

[0021] Preferably, the merged data set in S2 is derived from the sampling results of multiple data sets, and such training data is combined into an all-weather data set;

[0022] The sampling results of the multiple data sets include 9000 images sampled from Snow100K, 1069 images sampled from Raindrop, and 9000 images from Outdoor-Rain;

[0023] The test data set in S2 uses the Test1 data set, the RainDrop test data set, and the Snow100k-L test set.

[0024] Preferably, the network architecture in S3 is FDFormer, and the FDFormer is used to obtain feature information of different weather images in weather-degraded images. The design process of the FDFormer at least includes:

[0025] A two-stage self-attention module is designed based on Transformer to obtain similar and different features between different weather images, and remove the extracted weather to better achieve the overall image restoration effect. In the process, DFF dual-phase feature fusion and two-channel gated feedforward module are used to make the model training effect more accurate and robust.

[0026] That is, FDFormer includes a two-stage self-attention module, a two-phase feature fusion module and a two-channel gated feedforward module, and the two-stage self-attention module, the two-phase feature fusion module and the two-channel gated feedforward module are the TSSA module, the DFF module and the DTFF module;

[0027] The two-stage self-attention module performs adaptive feature refinement by sequentially inferring attention maps in the channel and spatial dimensions, and then multiplying these attention maps by the input feature map;

[0028] The dual-temporal feature fusion module uses convolution kernels of different sizes to capture multi-scale features of the image. The multi-scale features include large convolution kernels to capture global features and small convolution kernels to focus on detail features and restore texture details. These features of different scales are better integrated through splicing of channel dimensions.

[0029] The two-channel gated feedforward module is integrated into FTB, which enriches the representation of multi-range features and helps the image restoration process. The two-channel gated feedforward module integrates two different multi-range and multi-scale deep convolution paths during the transmission process;

[0030] The two-stage self-attention module, the dual-phase feature fusion module and the two-channel gated feedforward module are combined to improve the performance and generalization ability of the model.

[0031] Preferably, the two-stage self-attention module takes the original degraded image as input, and the specific implementation of image restoration is completed based on Transformer. The module combines channel attention and spatial attention sequentially to perform two-stage refinement on the input features;

[0032] The application process of the two-stage self-attention module includes at least the following steps:

[0033] The two-stage self-attention module first compresses the spatial dimension of the input feature map, and uses average pooling and maximum pooling operations to capture different spatial context descriptors, which are fed into a shared multi-layer perceptron (MLP) to produce a channel attention map;

[0034] Then, average pooling and maximum pooling operations are applied along the channel axis to concatenate the context descriptors to generate an efficient feature descriptor;

[0035] Apply a convolutional layer on the feature descriptor to generate a spatial attention map;

[0036] Preferably, the dual-phase feature fusion module replaces the multi-head self-attention in the transformer with a multi-scale parallel convolution kernel module, and on this basis proposes a multi-scale parallel large convolution kernel algorithm based on the multi-scale parallel large convolution kernel module and a multi-scale parallel large convolution kernel module;

[0037] The dual-temporal feature fusion module utilizes the multi-scale characteristics and large receptive field of the multi-scale parallel large convolution kernel module to focus the large convolution kernel on the global features and capture the areas with large blur. At the same time, an enhanced parallel attention module BPA structure is designed. This module can jointly use channel attention and pixel attention in parallel to extract the shared global information and position-related local information of the original features, so that it can effectively handle uneven weather distribution.

[0038] The dual-phase feature fusion module has both multi-scale characteristics and a large receptive field;

[0039] The application process of the dual-phase feature fusion module includes at least the following steps:

[0040] First, let x be the original feature map, and pass x b =BatchNorm(x) uses BatchNorm to normalize it. BatchNorm is used to accelerate network convergence, improve generalization ability and prevent overfitting. See the following formula:

[0041]

[0042] x2=Conv(x1),

[0043] x3=Concat(DWDConv19(x2),DWDConv13(x2),DWDConv7(x2))#(1)

[0044] Among them: PWConv represents point-by-point convolution, Conv represents convolution with kernel size = 5; DWDConv19 represents dilated convolution with kernel size = 19 and 7×7 depth-wise dilated convolution with dilation rate 3; DWDConv13 represents dilated convolution with kernel size = 13 and 5×5 depth-wise dilated convolution with dilation rate 3; DWDConv7 represents dilated convolution with kernel size 7 and 3×3 depth-wise dilated convolution; dilation rate is 3; Concat represents concatenating features in the channel dimension;

[0045] Three parallel dilated convolutions with different kernel sizes are used to extract multi-scale features, where large and medium dilated convolutions have long-range modeling capabilities and the large receptive field of self-attention in Transformer, which can focus on larger blurred areas;

[0046] The small dilated convolution can focus on a smaller blurred area, and the value of the restored texture detail x3 becomes three times x, see the following formula:

[0047] y=x+PWConv(GELU(PWConv(x3)))#(2)

[0048] Then, x3 is fed into a multilayer perceptron, which transforms the feature dimension of x3 to be the same as x. The multilayer perceptron contains two point-wise convolutions and uses GELU as the activation function;

[0049] Finally, the output of the multilayer perceptron is added to the identification shortcut x. The multilayer perceptron not only combines the three different types of features, but also plays a certain fitting role in the denoising features.

[0050] Preferably, the parallel attention module BPA structure mixes different types of attention mechanisms, and the parallel attention module BPA structure includes a simple pixel attention, a channel attention and a pixel attention;

[0051] The application process of the BPA structure includes at least the following steps:

[0052] Let x be the feature map, normalize it using BatchNorm by x=BatchNorm(x);

[0053] Pixel attention effectively extracts location-related information features with different haze distributions. The simple pixel attention module consists of two branches: PF s and PA s ;

[0054] PF s It is the feature extraction branch and the pixel gate branch;

[0055] Using PA sAs the pixel gating signal PF s , see the following formula:

[0056]

[0057] Among them: PWConv represents point-by-point convolution, Conv represents convolution with kernel size = 3, and pixel attention includes PA p Branch, used to extract global pixel gating features, see the following formula:

[0058]

[0059] Here, PWConv-GELU-PWConv is used to fit the features, the Sigmoid function is used to extract the global pixel gating features, and then PA p As the global pixel gating signal of bx;

[0060] Channel attention effectively extracts global information and changes the channel dimension of features. Channel attention has a CA c Branch, extract the features of the entire channel, see the following formula:

[0061]

[0062] Use global average pooling (GAP), PWConv-GELUPWConv and Sigmoid function to extract global channel gating features, and then use CA c As the global channel gating signal of bx, refer to the following formula:

[0063] F = Concat(F s ,F c ,F p ),y=x+PWConv(GELU(PWConv(F)))#(6)

[0064] Concatenate the three different attention gate results along the channel dimension, then apply an MLP with PWConv-GELU-PWConv to reduce the concatenated feature channel dimension to the same dimension as the input x, and finally add the output of the MLP with the identity shortcut bx;

[0065] The enhanced parallel attention module is more suitable for image restoration. The atmospheric light A is a shared global variable, while t(x) is a position-dependent local variable.

[0066] Channel attention can better extract shared global information and encode A, and pixel attention can better extract position-related information and encode t(x). It is believed that by extracting position-related and shared global information from the original features at the same time, the global optimization of the attention mechanism can be achieved;

[0067] However, when two different attention mechanisms are used in tandem, when channel attention modifies the original features by extracting global information and then pixel attention extracts position-dependent information of the modified features, the global optimal condition is not achieved;

[0068] In order to enable the enhanced attention module to simultaneously extract the shared global variables and position-related local variables of the original features, three different attention modules are processed in parallel. The original features A and t(x) are encoded in parallel, and the three independent attention results are connected to obtain the combined feature F, which is then fused through a multi-layer perceptron. It is believed that this parallel attention module BPA structure can better remove fuzzy features.

[0069] Preferably, the loss function designed for reconstructing the image in S4 at least comprises the following steps:

[0070] Using the restored high quality image I hq and the real image I gt The L1 norm of the pixel-level difference between is used as the reconstruction loss, see the following formula:

[0071]

[0072] In addition, note that Only the pixel-level similarity between the restored image and the real image is adjusted, while the block-level linear correlation is ignored;

[0073] The intensity relationships inherent in the images are disrupted by consistent patterns of weather-induced degradation;

[0074] By simulating the intensity relations in the ground truth, the degraded pixels are forced to occupy their original positions according to the original intensity ranking;

[0075] Therefore, the Pearson correlation between images is introduced as a means to adjust the linear relationship, see the following formula:

[0076]

[0077] in Represents the i-th pixel of the image;

[0078] andσ(I {·} ) represent the mean and standard deviation of the image sequence, respectively, and its value falls within the range of [-1,1] when the two images show perfect correlation;

[0079] The value of the function ρ reaches a value of 1, while in case of negative correlation, the function ρ reaches a value of -1;

[0080] Therefore, the relevant loss is expressed as follows:

[0081]

[0082] When the restored image is perfectly aligned with the ground truth, Therefore, the overall loss function is defined as:

[0083]

[0084] where α is the weight of the associated loss.

[0085] Compared with the prior art, the present invention has the following beneficial effects:

[0086] 1. Compared with traditional image restoration methods, the present invention does not require artificial prior knowledge. Relying on the powerful learning ability of neural networks, the model has strong performance and more accurate restoration results in the task of restoring weather-degraded images.

[0087] 2. Compared with other deep learning image restoration methods, the present invention is reasonably designed and expanded based on Transformer. The proposed FDFormer network can use channel and spatial two-stage attention to allow the model to more comprehensively capture feature information and enhance details, thereby enhancing the robustness of the model for different weather and different data sets. Compared with other excellent weather-degraded image restoration methods, it has better performance and generalization ability; it also has good performance for challenges such as unlearned harsh environments; and it significantly reduces the number of model parameters and reduces the possibility of overfitting, meeting lightweight requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0088] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for describing the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0089] Figure 1 A flow chart of the weather-degraded image restoration method provided by the present invention;

[0090] Figure 2 A schematic diagram of the FDFormer model provided by the present invention;

[0091] Figure 3 A schematic diagram of a TSSA module provided by the present invention;

[0092] Figure 4 A schematic diagram of the DTFF module provided by the present invention;

[0093] Figure 5 Visual comparison chart of image weather removal of different models provided by the present invention on dataset D. DETAILED DESCRIPTION

[0094] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0095] See also Figure 1 , an all-in-one method for bad weather removal by fusing bi-temporal features with a two-stage converter, comprises at least the following steps:

[0096] S1: Batch preprocessing of degraded images in the dataset;

[0097] S2: Merge the original dataset with the synthetic dataset to perform non-overlapping dataset division;

[0098] S3: Design a network architecture for feature extraction and image restoration;

[0099] S4: Design a loss function for reconstructing images to improve image similarity;

[0100] S5: Use the trained model to perform measurement verification, and the model training is completed;

[0101] S6: Restoration is completed by inputting the image to be degraded into the model.

[0102] In S1, batch preprocessing of the degraded images in the data set includes at least the following steps:

[0103] Randomly crop the original degraded image, and correspond the corresponding labels to the randomly cropped images to ensure that the labels and coordinate cropped images have the same size;

[0104] The images and labels are enhanced, and then converted into tensors through the conversion channel. The processed images and labels are used for model training.

[0105] The merged dataset in S2 comes from the sampling results of multiple datasets, and this training data is combined into an all-weather dataset;

[0106] The sampling results of multiple datasets include 9000 images sampled from Snow100K, 1069 images sampled from Raindrop, and 9000 images from Outdoor-Rain;

[0107] The test datasets in S2 use the Test1 dataset, the RainDrop test dataset, and the Snow100k-L test set.

[0108] The network architecture in S3 is FDFormer, which is used to obtain the feature information of different weather images in weather-degraded images. The design process of FDFormer at least includes:

[0109] A two-stage self-attention module is designed based on Transformer to obtain similar and different features between different weather images, and remove the extracted weather to better achieve the overall image restoration effect. In the process, DFF dual-phase feature fusion and two-channel gated feedforward module are used to make the model training effect more accurate and robust.

[0110] That is, FDFormer includes a two-stage self-attention module, a two-phase feature fusion module and a two-channel gated feedforward module, which are the TSSA module, the DFF module and the DTFF module.

[0111] FDFormer structure reference Figure 2 ;

[0112] The two-stage self-attention module performs adaptive feature refinement by sequentially inferring attention maps in the channel and spatial dimensions, and then multiplying these attention maps by the input feature map;

[0113] The dual-temporal feature fusion module uses convolution kernels of different sizes to capture the multi-scale features of the image. The multi-scale features include large convolution kernels to capture global features and small convolution kernels to focus on detail features and restore texture details. These features of different scales are better integrated through splicing of channel dimensions.

[0114] like Figure 4 As shown, two-channel gated feedforwards have previously typically utilized single-range or single-scale convolutions in standard feedforward networks to support local context. Nevertheless, these methods often ignore the correlation between dynamically distributed weather-induced degradations. In practice, multi-scale information can be extracted not only by enlarging the kernel size but also by utilizing the dilation mechanism;

[0115] The two-channel gated feedforward module is integrated into FTB to enrich the representation of multi-range features and help the image restoration process. The two-channel gated feedforward module integrates two different multi-range and multi-scale deep convolution paths during the transmission process;

[0116] The two-stage self-attention module, the dual-phase feature fusion module and the two-channel gated feedforward module are combined to improve the performance and generalization ability of the model.

[0117] See also Figure 3The two-stage self-attention module takes the original degraded image as input. The specific implementation of image restoration is based on Transformer. This module combines channel attention and spatial attention sequentially to refine the input features in two stages.

[0118] The application process of the two-stage self-attention module includes at least the following steps:

[0119] The two-stage self-attention module first compresses the spatial dimension of the input feature map, and uses average pooling and maximum pooling operations to capture different spatial context descriptors, which are fed into a shared multi-layer perceptron (MLP) to produce a channel attention map;

[0120] Then, average pooling and maximum pooling operations are applied along the channel axis to concatenate the context descriptors to generate an efficient feature descriptor;

[0121] Apply a convolutional layer on the feature descriptor to generate a spatial attention map;

[0122] This design allows the model to focus on "which channels are important" first, and then focus on "which positions in space are important", so as to more comprehensively capture the key information in the features and adaptively adjust the importance of each channel and spatial position in the input feature map. At the same time, the design of the TSSA module takes computational efficiency into consideration, simplifies the calculation of channel attention through global average pooling and maximum pooling, and uses simple convolution operations to achieve spatial attention. These designs enable the TSSA module to maintain a low additional computational cost while bringing performance improvements.

[0123] The dual-phase feature fusion module replaces the multi-head self-attention in the transformer with a multi-scale parallel convolution kernel module, and on this basis, a multi-scale parallel large convolution kernel algorithm based on the multi-scale parallel large convolution kernel module and a multi-scale parallel large convolution kernel module are proposed;

[0124] The dual-temporal feature fusion module utilizes the multi-scale characteristics and large receptive field of the multi-scale parallel large convolution kernel module to focus the large convolution kernel on the global features and capture the areas with large blur. At the same time, an enhanced parallel attention module BPA structure is designed. This module can jointly use channel attention and pixel attention in parallel to extract the shared global information and location-related local information of the original features, so that it can effectively handle uneven weather distribution.

[0125] The dual-phase feature fusion module has both multi-scale characteristics and a large receptive field;

[0126] The application process of the dual-phase feature fusion module includes at least the following steps:

[0127] First, let x be the original feature map, and pass x b =BatchNorm(x) uses BatchNorm to normalize it. BatchNorm is used to accelerate network convergence, improve generalization ability and prevent overfitting. See the following formula:

[0128]

[0129] x2=Conv(x1),

[0130] x3=Concat(DWDConv19(x2),DWDConv13(x2),DWDConv7(x2))#(1)

[0131] Among them: PWConv represents point-by-point convolution, Conv represents convolution with kernel size = 5; DWDConv19 represents dilated convolution with kernel size = 19 and 7×7 depth-wise dilated convolution with dilation rate 3; DWDConv13 represents dilated convolution with kernel size = 13 and 5×5 depth-wise dilated convolution with dilation rate 3; DWDConv7 represents dilated convolution with kernel size 7 and 3×3 depth-wise dilated convolution; dilation rate is 3; Concat represents concatenating features in the channel dimension;

[0132] Three parallel dilated convolutions with different kernel sizes are used to extract multi-scale features, where large and medium dilated convolutions have long-range modeling capabilities and the large receptive field of self-attention in Transformer, which can focus on larger blurred areas;

[0133] The small dilated convolution can focus on a smaller blurred area, and the value of the restored texture detail x3 becomes three times x, see the following formula:

[0134] y=x+PWConv(GELU(PWConv(x3)))#(2)

[0135] Then, x3 is fed into a multilayer perceptron, which transforms the feature dimension of x3 to be the same as x. The multilayer perceptron contains two point-wise convolutions and uses GELU as the activation function;

[0136] Finally, the output of the multilayer perceptron is added to the identification shortcut x. The multilayer perceptron not only combines the three different types of features, but also plays a certain fitting role in the denoising features.

[0137] The parallel attention module BPA structure mixes different types of attention mechanisms. The parallel attention module BPA structure includes a simple pixel attention, a channel attention, and a pixel attention.

[0138] The application process of the BPA structure includes at least the following steps:

[0139] Let x be the feature map, normalize it using BatchNorm by x=BatchNorm(x);

[0140] Pixel attention effectively extracts location-related information features with different haze distributions. The simple pixel attention module consists of two branches: PF s and PA s ;

[0141] PF s It is the feature extraction branch and the pixel gate branch;

[0142] Using PA s As the pixel gating signal PF s , see the following formula:

[0143]

[0144] Among them: PWConv represents point-by-point convolution, Conv represents convolution with kernel size = 3, and pixel attention includes PA p Branch, used to extract global pixel gating features, see the following formula:

[0145]

[0146] Here, PWConv-GELU-PWConv is used to fit the features, the Sigmoid function is used to extract the global pixel gating features, and then PA p As the global pixel gating signal of bx;

[0147] Channel attention effectively extracts global information and changes the channel dimension of features. Channel attention has a CA c Branch, extract the features of the entire channel, see the following formula:

[0148]

[0149] Use global average pooling (GAP), PWConv-GELUPWConv and Sigmoid function to extract global channel gating features, and then use CA c As the global channel gating signal of bx, refer to the following formula:

[0150] F = Concat(F s ,F c ,F p ),y=x+PWConv(GELU(PWConv(F)))#(6)

[0151] Concatenate the three different attention gate results along the channel dimension, then apply an MLP with PWConv-GELU-PWConv to reduce the concatenated feature channel dimension to the same dimension as the input x, and finally add the output of the MLP with the identity shortcut bx;

[0152] The enhanced parallel attention module is more suitable for image restoration. The atmospheric light A is a shared global variable, while t(x) is a position-dependent local variable.

[0153] Channel attention can better extract shared global information and encode A, and pixel attention can better extract position-related information and encode t(x). It is believed that by extracting position-related and shared global information from the original features at the same time, the global optimization of the attention mechanism can be achieved;

[0154] However, when two different attention mechanisms are used in tandem, when channel attention modifies the original features by extracting global information and then pixel attention extracts position-dependent information of the modified features, the global optimal condition is not achieved;

[0155] In order to enable the enhanced attention module to simultaneously extract the shared global variables and position-related local variables of the original features, three different attention modules are processed in parallel. The original features A and t(x) are encoded in parallel, and the three independent attention results are connected to obtain the combined feature F, which is then fused through a multi-layer perceptron. It is believed that this parallel attention module BPA structure can better remove fuzzy features.

[0156] In S4, we overcome the problem of adjusting the pixel-level similarity between the restored image and the real image while ignoring the block-level linear correlation. Considering that the inherent intensity relationship in the image is destroyed by the consistent pattern of weather-induced degradation, we force the degraded pixels to occupy their original positions according to the original intensity ranking. We introduce the Pearson correlation between images as a means of adjusting the linear relationship;

[0157] The Pearson correlation between images is introduced as a means to adjust the linear relationship, see the following formula:

[0158]

[0159] in Represents the i-th pixel of the image;

[0160] andσ(I {·} ) represent the mean and standard deviation of the image sequence, respectively, and its value falls within the range of [-1,1] when the two images show perfect correlation;

[0161] The value of the function ρ reaches a value of 1, while in case of negative correlation, the function ρ reaches a value of -1;

[0162] Therefore, the relevant loss is expressed as follows:

[0163]

[0164] When the restored image is perfectly aligned with the ground truth, Therefore, the overall loss function is defined as:

[0165]

[0166] where α is the weight of the associated loss.

[0167] The calculation principle of the image restoration evaluation index involved in S5 is as follows: Use the structural measurement index (SSIM) and the peak signal-to-noise ratio (PSNR) to measure the image quality. SSIM can measure the degree of distortion of the image, and can also measure the similarity between two images. From the perspective of image composition, structural information is defined as an attribute that reflects the structure of objects in the scene independent of brightness and contrast, and distortion is modeled as a combination of three different factors: brightness, contrast, and structure. The mean is used as an estimate of brightness, the standard deviation is used as an estimate of contrast, and the covariance is used as a measure of structural similarity. The larger the value, the more similar the two images are, which means the better the effect of restoring the image. PSNR measures the quality of the image after denoising or adding noise. The larger the value, the better the filtering effect. In the image, that is, the distortion of the image, the higher the PSNR value, the smaller the image distortion, indicating that the image quality is excellent and very close to the original image.

[0168] PSNR is defined based on MSE (mean square error). Given an original image I of size m*n and a noisy image K after adding noise to it, its MSE can be defined as:

[0169]

[0170] Where MAXI is the maximum pixel value of the image, the unit of PSNR is dB, and if each pixel is represented by 8 bits of binary, its value is 2^8-1=255.

[0171] But please note that this is a calculation method for grayscale images. If it is a color image, it can usually be calculated by the following method: calculate the MSE value of each channel of the three channels of the RGB image and then find the average value, and then find the PSNR.

[0172] Directly use the built-in function psnr() of Matlab (note that this function treats all images as grayscale images), convert the image to YCbCr format, and only calculate the PSNR of the Y component, that is, the brightness component.

[0173]

[0174] Luminance measurement - actually μ x , μ y Function

[0175]

[0176] Contrast measurement—actually σ x and σ y Function

[0177]

[0178] Structural measurement—actually σ x σ y With σ x ,σ y Function

[0179]

[0180] The final structural similarity formula is as follows:

[0181] SSIM(x,y)=[l(x,y)] α [c(x,y)] β ·[s(x,y)] γ #(17)

[0182] The three parameters α, β, and γ are used to indicate the importance of the three modules, and SSIM represents the similarity. This operator should satisfy the basic properties as a metric.

[0183] Based on the above content, the following specific application is proposed:

[0184] The present invention is based on the public rain, snow and fog weather dataset, including 9,000 images sampled from Snow100K, 1,069 images sampled from Raindrop and 9,000 images from Outdoor-Rain. This combination of training data is called the all-weather dataset. For the test dataset, we use the Test1 dataset, the RainDrop test dataset and the Snow100k-L test set, on which we train and verify the effectiveness of the model and method. All compared models use public source code and the same training and testing methods as this method. When measuring performance, the following indicators are used to evaluate the measurement model: PSNR (peak signal-to-noise ratio) and SSIM (structural measurement index).

[0185] The method proposed in this invention is compared with many excellent image restoration methods. The verification results on three combined data sets are shown in Tables 1, 2 and 3. The visual comparison results are shown in Tables 1, 2 and 3. Figure 5 It can be seen that the proposed model can better restore weather-degraded images in severe weather conditions, with better accuracy and robustness. It is worth emphasizing that our model shows excellent performance on the largest validation dataset, which shows that the model has excellent generalization performance. All experiments show that the proposed model achieves the best restoration effect in weather removal of contrasting images.

[0186] Table 1 Comparison results of image restoration on dataset A (optimal underline)

[0187]

[0188] Table 2 Comparison results of image restoration on dataset B (optimal underline)

[0189]

[0190] Table 3 Comparison results of image restoration on dataset C (optimal underline)

[0191]

[0192] In summary, the present invention aims at the limitations of most Transformer-based image weather removal methods in self-attention mechanism, excessive computational burden, insufficient utilization of global information, limited coverage of existing severe weather datasets, and poor training results. This patent proposes a new fast, efficient, and accurate image weather removal model FDFormer. FDFormer is a general network framework with a novel transformer block that uses a single model example to solve all severe weather removal tasks. Among them, a two-stage transformer fusion dual-temporal feature network is proposed to restore images affected by severe weather. It is powered by a mechanism called two-stage self-attention, which will sequentially infer attention maps in two dimensions (channel and space) and then multiply these attention maps by the input feature map for adaptive feature refinement, which can solve the problem of lack of utilization of spatial and local information in previous methods. At the same time, we introduce dual-temporal feature fusion, using large convolution kernels to capture global features and foggy areas, and small convolution kernels to focus on detail features and restore texture details, so as to better fuse features. We also observed that the common pixel loss ignores the linear association and correlation between the output and the true value. Therefore, we propose to use the Pearson correlation coefficient as the loss function to enforce the restored pixels to follow the same order as the ground truth. In order to demonstrate the excellent performance of the proposed segmentation model on the semantic segmentation task, multiple datasets are used for training to expand the applicability of the method.

[0193] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

Claims

1. An all-in-one method for bad weather removal with two-phase converters integrating dual-phase features, characterized by: At least the following steps are included: S1: Batch preprocessing of degraded images in the dataset; S2: Merge the original dataset with the synthetic dataset to perform non-overlapping dataset division; S3: Design a network architecture for feature extraction and image restoration; S4: Design a loss function for reconstructing images to improve image similarity; S5: Use the trained model to perform measurement verification, and the model training is completed; S6: Restoration is completed by inputting the image to be degraded into the model.

2. The all-in-one method for removing bad weather by integrating dual-time-phase features with a two-stage converter according to claim 1, characterized in that: The batch preprocessing of the degraded images in the data set in S1 at least includes the following steps: Randomly crop the original degraded image, and correspond the corresponding labels to the randomly cropped images to ensure that the labels and coordinate cropped images have the same size; The images and labels are enhanced, and then converted into tensors through the conversion channel. The processed images and labels are used for model training.

3. The all-in-one method for removing bad weather by integrating dual-time-phase characteristics with a two-stage converter according to claim 1, characterized in that: The merged data set in S2 is derived from the sampling results of multiple data sets, and such training data is combined into an all-weather data set; The test data set in S2 uses the Test1 data set, the RainDrop test data set, and the Snow100k-L test set.

4. The all-in-one method for removing bad weather by integrating dual-phase characteristics with a two-stage converter according to claim 1, characterized in that: The network architecture in S3 is FDFormer, which is used to obtain feature information of different weather images in weather-degraded images. The design process of FDFormer at least includes: A two-stage self-attention module is designed based on Transformer to obtain similar and different features between different weather images, and remove the extracted weather to better achieve the overall image restoration effect. In the process, DFF dual-phase feature fusion and two-channel gated feedforward module are used to make the model training effect more accurate and robust. That is, FDFormer includes a two-stage self-attention module, a two-phase feature fusion module and a two-channel gated feedforward module, and the two-stage self-attention module, the two-phase feature fusion module and the two-channel gated feedforward module are the TSSA module, the DFF module and the DTFF module; The two-stage self-attention module performs adaptive feature refinement by sequentially inferring attention maps in the channel and spatial dimensions, and then multiplying these attention maps by the input feature map; The dual-temporal feature fusion module uses convolution kernels of different sizes to capture multi-scale features of the image. The multi-scale features include large convolution kernels to capture global features and small convolution kernels to focus on detail features and restore texture details. These features of different scales are better integrated through splicing of channel dimensions. The two-channel gated feedforward module is integrated into FTB, which enriches the representation of multi-range features and helps the image restoration process. The two-channel gated feedforward module integrates two different multi-range and multi-scale deep convolution paths during the transmission process; The two-stage self-attention module, the dual-phase feature fusion module and the two-channel gated feedforward module are combined to improve the performance and generalization ability of the model.

5. The all-in-one method for removing bad weather by fusing dual-time-phase features with a two-stage converter according to claim 4, characterized in that: The two-stage self-attention module takes the original degraded image as input. The specific implementation of image restoration is completed based on Transformer. The module combines channel attention and spatial attention sequence to perform two-stage refinement on the input features. The application process of the two-stage self-attention module includes at least the following steps: The two-stage self-attention module first compresses the spatial dimension of the input feature map, and uses average pooling and maximum pooling operations to capture different spatial context descriptors, which are fed into a shared multi-layer perceptron to produce a channel attention map. Then, average pooling and maximum pooling operations are applied along the channel axis to concatenate the context descriptors to generate an efficient feature descriptor; A convolutional layer is applied on the feature descriptors to generate a spatial attention map.

6. The all-in-one method for removing bad weather by fusing dual-time-phase features with a two-stage converter according to claim 4, characterized in that: The dual-phase feature fusion module replaces the multi-head self-attention in the transformer with a multi-scale parallel convolution kernel module, and on this basis proposes a multi-scale parallel large convolution kernel algorithm based on the multi-scale parallel large convolution kernel module and a multi-scale parallel large convolution kernel module; The dual-temporal feature fusion module utilizes the multi-scale characteristics and large receptive field of the multi-scale parallel large convolution kernel module to focus the large convolution kernel on the global features and capture the areas with large blur. At the same time, an enhanced parallel attention module BPA structure is designed. This module can jointly use channel attention and pixel attention in parallel to extract the shared global information and position-related local information of the original features, so that it can effectively handle uneven weather distribution. The dual-phase feature fusion module has both multi-scale characteristics and a large receptive field; The application process of the dual-phase feature fusion module includes at least the following steps: First, let x be the original feature map, and pass x b =BatchNorm(x) uses BatchNorm to normalize it. BatchNorm is used to accelerate network convergence, improve generalization ability and prevent overfitting. See the following formula: x2=Conv(x1), x3 = Concat (DWDConv19 (x2), DWDConv13 (x2), DWDConv7 (x2)) # (1) Where: PWConv represents point-by-point convolution, Conv represents convolution with kernel size = 5; DWDConv19 represents dilated convolution with kernel size = 19 and a 7×7 depth-wise dilated convolution with dilation rate 3; DWDConv13 represents dilated convolution with kernel size = 13 and a 5×5 depth-wise dilated convolution with dilation rate 3; DWDConv7 represents dilated convolution with kernel size 7 and a 3×3 depth-wise dilated convolution; the dilation rate is 3; Concat represents concatenating features in the channel dimension; Three parallel dilated convolutions with different kernel sizes are used to extract multi-scale features, where large and medium dilated convolutions have long-range modeling capabilities and the large receptive field of self-attention in Transformer, which can focus on larger blurred areas; The small dilated convolution can focus on a smaller blurred area, and the value of the restored texture detail x3 becomes three times x, see the following formula: y=x+PWConv(GELU(PWConv(x3)))#(2) Then, x3 is fed into a multilayer perceptron, which transforms the feature dimension of x3 to be the same as x. The multilayer perceptron contains two point-wise convolutions and uses GELU as the activation function; Finally, the output of the multilayer perceptron is added to the identification shortcut x. The multilayer perceptron not only combines the three different types of features, but also plays a certain fitting role in the denoising features.

7. The all-in-one method for removing bad weather by fusing dual-time-phase characteristics with a two-stage converter according to claim 6, characterized in that: The parallel attention module BPA structure mixes different types of attention mechanisms, including a simple pixel attention, a channel attention and a pixel attention; The application process of the BPA structure includes at least the following steps: Let x be the feature map, normalize it using BatchNorm by x=BatchNorm(x); Pixel attention effectively extracts location-related information features. Different haze distributions. A simple pixel attention module consists of two branches: PF s and PA s ; PF s It is the feature extraction branch and the pixel gate branch; Using PA s As the pixel gating signal PF s , see the following formula: Among them: PWConv represents point-by-point convolution, Conv represents convolution with kernel size = 3, and pixel attention includes PA p Branch, used to extract global pixel gating features, see the following formula: Here, PWConv-GELU-PWConv is used to fit the features, the Sigmoid function is used to extract the global pixel gating features, and then PA p As the global pixel gating signal of bx; Channel attention effectively extracts global information and changes the channel dimension of features. Channel attention has a CA c Branch, extract the features of the entire channel, see the following formula: Use global average pooling (GAP), PWConv-GELUPWConv and Sigmoid function to extract global channel gating features, and then use CA c As the global channel gating signal of bx, refer to the following formula: F=Concat(F s ,F c ,F p ),y=x+PWConv(GELU(PWConv(F)))#(6) Concatenate the three different attention gate results along the channel dimension, then apply an MLP with PWConv-GELU-PWConv to reduce the concatenated feature channel dimension to the same dimension as the input x, and finally add the output of the MLP with the identity shortcut bx; The enhanced parallel attention module is more suitable for image restoration. The atmospheric light A is a shared global variable, while t(x) is a position-dependent local variable. Channel attention can better extract shared global information and encode A, and pixel attention can better extract position-related information and encode t(x). It is believed that by extracting position-related and shared global information from the original features at the same time, the global optimization of the attention mechanism can be achieved; However, when two different attention mechanisms are used in tandem, when channel attention modifies the original features by extracting global information and then pixel attention extracts position-dependent information of the modified features, the global optimal condition is not achieved; In order to enable the enhanced attention module to simultaneously extract the shared global variables and position-related local variables of the original features, three different attention modules are processed in parallel. The original features A and t(x) are encoded in parallel, and the three independent attention results are connected to obtain the combined feature F, which is then fused through a multi-layer perceptron. It is believed that this parallel attention module BPA structure can better remove fuzzy features.

8. The all-in-one method for removing bad weather by integrating dual-time-phase characteristics with a two-stage converter according to claim 1, characterized in that: The loss function designed in S4 for reconstructing the image comprises at least the following steps: Using the restored high quality image I hq and the real image I gt The L1 norm of the pixel-level difference between is used as the reconstruction loss, see the following formula: In addition, note that Only the pixel-level similarity between the restored image and the real image is adjusted, while the block-level linear correlation is ignored; The intensity relationships inherent in the images are disrupted by consistent patterns of weather-induced degradation; By simulating the intensity relations in the ground truth, the degraded pixels are forced to occupy their original positions according to the original intensity ranking; Therefore, the Pearson correlation between images is introduced as a means to adjust the linear relationship, see the following formula: in Represents the i-th pixel of the image; andσ(I {·} ) represent the mean and standard deviation of the image sequence, respectively, and its value falls within the range of [-1,1] when the two images show perfect correlation; The value of the function ρ reaches a value of 1, while in case of negative correlation, the function ρ reaches a value of -1; Therefore, the relevant loss is expressed as follows: When the restored image is perfectly aligned with the ground truth, Therefore, the overall loss function is defined as: where α is the weight of the associated loss.

Images