Image rain and fog combined removal method based on multi-stage encoder-decoder structure

Abstract

The invention relates to the field of image rain and fog combined removal and deep learning, and provides an image rain and fog combined removal method based on a multi-stage encoder-decoder structure, the method comprises an encoder and a decoder of five stages, the encoder is composed of a refocusing convolution module, a cavity convolution module and a space and channel reconstruction convolution module, and the cavity convolution module is composed of a refocusing convolution module, a cavity convolution module and a space and channel reconstruction convolution module. The method aims at extracting rich local semantics and high-frequency information, and calibrating, filtering and reserving channel information with importance and representativeness in a feature map through a channel attention mechanism. The decoder is composed of a Swin Transform and a supervision attention module, and aims at extracting global low-frequency features and aggregating context information, and generating a feature map rich in wide space context and significant deep representation. The rain and fog removal image generated by the method is more in line with human eye perception, and the problems of artifact noise, color distortion and the like of an existing image rain and fog combined removal method can be effectively solved.

Description

Technical Field

[0001] The present invention relates to the technical field of image restoration, and in particular to a method for jointly removing rain and fog from an image based on a multi-stage encoder-decoder structure. Background Art

[0002] Image deraining and dehazing aims to improve image clarity by effectively removing the effects of rain and fog. In addition, the color of light changes during the scattering and absorption of rain and fog, which further reduces the ability of the human eye to recognize colors. If the input images for high-level vision tasks are of low quality and contain a lot of rain and fog, they may mislead the judgment of related systems, such as real-time monitoring and intelligent driving. Therefore, image deraining and dehazing are considered to be meaningful low-level vision tasks that can effectively improve the reliability of the model under degraded conditions.

[0003] Removing rain and fog from images is very important for improving visual effects affected by weather factors. Researchers have proposed a variety of methods for removing rain and fog, which can be mainly divided into two categories: traditional image restoration methods based on physical simulation and image restoration methods using deep learning. Traditional techniques usually rely on physical simulation to simulate the image degradation process. However, data-driven end-to-end trainable methods no longer require complex physical models. These methods have higher flexibility and can adapt to a variety of different environmental conditions. By adopting a data-centric architecture, the model is better at handling situations where weather effects vary in real life.

[0004] With the continuous progress of machine learning, image restoration techniques based on deep convolutional neural networks (CNNs) have gradually come to the fore. These CNN methods are different from traditional methods, which rely on manual priors, while CNN methods learn data-driven mapping relationships between different variables. They are able to predict potential non-degraded images or calculate the residual difference between degraded and non-degraded images, thereby helping to determine the parameters of nonlinear models. However, the convolution operation in deep CNNs has certain limitations in terms of receptive field, which makes deep CNNs more suitable for processing details and textures in images, such as edges, corners, and other local features that reflect rapidly changing areas of the image. In addition, the convolution filters of deep CNNs cannot perform dynamic reasoning. In contrast, the visual transformer (ViT) has become a more powerful alternative with dynamic reasoning capabilities.

[0005] Since ViT-based models have surpassed almost all CNN architectures in high-level vision tasks, many enhanced ViT variants have emerged to further leverage their advantages. A large number of studies have shown that ViT models perform well and efficiently in various vision tasks, including image classification, object detection, and semantic segmentation. These models have significant advantages in capturing global context and long-range dependencies, and are able to handle larger structures or global information in images, such as the overall shape of objects, scene layout, etc. Despite these advances, there is still a large gap in the application of ViT models in the task of joint rain and fog removal in images. Only a few ViT-based models have achieved state-of-the-art performance in this area, indicating that further exploration and development are still needed to fully tap the potential of ViT architectures in these complex image restoration tasks.

[0006] The present invention further studies a method specifically used for the joint removal of rain and fog from images, aiming to more efficiently convert rain and fog mixed images into clear and clean rain and fog-free images end-to-end. Summary of the invention

[0007] Aiming at the problems of serious artifact noise and color distortion in existing image rain and fog removal methods, and the fact that these problems may lead to model parameter estimation deviation, thus obtaining poor rain and fog removal effect, the present invention proposes an image rain and fog removal method based on a multi-stage encoder-decoder structure. The method adopts a five-stage structure to decompose the complex image restoration task into simpler subtasks. Image features are transferred and converted between each stage, thereby improving the effect of rain and fog removal.

[0008] In order to achieve the above object, the present invention adopts the following technical solution:

[0009] A method for jointly removing rain and fog from an image based on a multi-stage encoder-decoder structure comprises the following steps:

[0010] Step 1: Obtain images without rain and fog and images with rain and fog, and divide them into training data sets and test data sets. The images without rain and fog can be obtained from a public data set, and the images with rain and fog can be generated by simulating the effects of rain and fog;

[0011] Step 2: Build a rain and fog joint removal network with a multi-stage encoder-decoder structure. The input of the rain and fog joint removal network is the image with rain and fog in step 1. The image with rain and fog is end-to-end jointly removed from rain and fog, and the clean image after removal is directly output.

[0012] Step 3: Use the training data set and loss function in step 1 to train the rain and fog joint removal network in step 2 until the maximum number of iterations is reached;

[0013] Step 4: For the image that needs to be jointly removed from rain and fog, input it into the rain and fog joint removal network obtained after training in step 3 to obtain the final rain and fog joint removal image.

[0014] Furthermore, the rainy and foggy image input in step 2 is denoted as I∈R H×W×C , where H represents the height of the image, W represents the width of the image, and C represents the number of channels of the image. Through the 3×3 convolution operation on the input, the shallow features are extracted, which is expressed as F0=H CoNV (I) The rain and fog joint removal network of the multi-stage encoder-decoder structure in step 2 includes five stages of encoders and decoders, wherein the encoder of each stage is composed of a refocusing convolution module, a hole convolution module, and a space and channel reconstruction convolution module to extract high-frequency information in the image, and the decoder uses a Swin Transformer module to extract low-frequency information in the image, followed by a supervised attention module to transfer and transform features between the two stages.

[0015] Furthermore, the encoder described in step 2 includes three parts: a refocusing convolution module, a dilated convolution module, and a spatial and channel reconstruction convolution module, which aims to extract rich local semantics and high-frequency information, calibrate, filter, and retain important and representative channel information in the feature map through the channel attention mechanism, where the encoder feature is represented as:

[0016]

[0017] In the formula Represents the features encoded by the i-th layer encoder. The encoder uses H E (·) indicates that the input of the encoder in the first three stages is the shallow feature F0, and the input of the encoder in the fourth and fifth stages is respectively and

[0018] Furthermore, the refocusing convolution module in the encoder can be reparameterized and refocused to enhance the high-frequency feature extraction capability of the network, as follows:

[0019] F RefConv =H CONV (X⊙H M (X)),

[0020] Where X represents the feature map input, H M (·) is a network for generating the focus matrix, and X⊙H M (X) is a pixel-wise weight redistribution operation. Refocused convolution leverages existing convolution kernels as priors and enhances model performance by learning new representations with trainable refocusing transformations.

[0021] Furthermore, the dilated convolution module in the encoder can capture multi-scale context information while expanding the receptive field without increasing the number of parameters. The formula is as follows:

[0022]

[0023] Where X represents the feature map input, r represents the expansion rate, and k is the index of the convolution kernel, ranging from k∈[-K, K]. Dilated convolutions provide a feasible solution by expanding the receptive field of the convolution kernel without increasing the number of parameters. This allows the network to better capture contextual information in the image without losing resolution.

[0024] Furthermore, the spatial and channel reconstruction convolutional module in the encoder can reduce the spatial and channel redundancy between features in the network and reduce the computational complexity, as shown in the following formula:

[0025] F SCConv =H CRU (H SRU (X)),

[0026] Where H SRU The main focus is on reducing spatial redundancy through a process called spatial segmentation. This process involves segmenting the features along the spatial dimension, performing convolution operations on each segment separately, and then reconstructing the output. CRU Mainly targeting channel redundancy, it effectively reduces the redundant information between channels through channel segmentation and channel reweighting operations.

[0027] F SRU =Concat(H CONV (X i )), i = 1, 2, ...,

[0028] Where X i It means dividing the input X into subsets according to the spatial dimension, then performing spatial parallel convolution on each subset in the spatial dimension, and finally reconstructing the spatial subsets, that is, reconstructing all the convolved subsets back to their original spatial dimensions. Concat means merging subsets.

[0029]

[0030] In the formula It means that the input X is divided into multiple channel dimensions, and then spatially parallel convolution is performed on the subsets in each spatial dimension. Finally, channel subset reconstruction is performed and all the convolved subsets are merged back to the original channel dimension.

[0031] Furthermore, the decoder described in step 2 is composed of Swi nTransform r It is composed of a supervised attention module, which aims to extract and aggregate contextual information for global low-frequency features and generate a feature map rich in extensive spatial context and significant deep representation. The decoder feature is represented as:

[0032]

[0033] In the formula Represents the features after decoding by the i-th layer decoder. The decoder uses H D (·)express, It represents the features encoded by the i-th layer encoder, Concat represents the connection operation, the input of the first stage decoder is the features encoded by the first stage encoder, and the input of the second to fifth stages is the features encoded by the encoder of the current stage and the features decoded by the decoder of the previous stage. The two parts are connected along the channel dimension to form a new tensor.

[0034] Furthermore, the Swin Transformer module in the decoder is used to extract global low-frequency features and aggregate context information. There are three feature tensors in Swin Transformer, namely Query (Q), Key (K) and Value (V). The formula is as follows:

[0035]

[0036] In the formula, Corresponding to a single window and head, M represents the window size, d represents the dimension, and P is the learnable relative position encoding. The model uses modified layer normalization, which differs from standard layer normalization (LayerNorm) in that an additional convolutional layer H is used in the module. RLN (·) to learn the transformation of input mean and standard deviation, thus helping the model capture more complex data distribution characteristics.

[0037]

[0038] Where μ and σ 2 Represent the mean and variance of the input X in the spatial and channel dimensions, respectively. Unlike layer normalization, rectified layer normalization normalizes individual samples in the feature dimension. This makes rectified layer normalization independent of batch size and more suitable for scenarios where batch size changes significantly or sequence length changes dynamically.

[0039] Furthermore, the supervised attention module in the decoder is capable of transferring and transforming features between the two stages. The input of the supervised attention module includes the feature map X generated by the Swin Transformer and the original input degraded image containing rain and fog. First, a 1×1 convolution operation transforms X into the spatial dimension of the original input image, changing its number of channels to 3. Then, this transformed feature map is added to the original degraded input image to produce the restored image of this stage. Subsequently, the restored image is processed by a convolution layer with a 1×1 convolution and a sigmoid activation function. This operation generates a pixel-level attention mask, which is the same size as X and indicates the importance of the corresponding pixel. After that, the attention mask is element-wise multiplied with the transformed feature map obtained by the convolution operation. This multiplication aims to calibrate each pixel in the feature map, enhance and emphasize high-value information, and suppress irrelevant details. Finally, by establishing a residual connection with X, the attention-enhanced feature is obtained, which is the output of the supervised attention module.

[0040] Furthermore, the clean image after the direct output removal described in step 2 is decoded by the final 5th stage decoder to obtain a feature map which is spatially transformed by the original resolution sub-network and then bitwise added to F0, where the expression is:

[0041]

[0042] Where O represents the output image after the model removes rain and fog, represents the feature map decoded by the 5th stage decoder, H OSR (·) represents the original resolution subnetwork, which is composed of a cascade of multiple original resolution modules for deep processing and enhancing input features. The original resolution module contains multiple channel attention modules and convolutional layers, which are responsible for feature extraction and information fusion.

[0043] Furthermore, the loss function described in step 3 includes the PSNR (Peak Signal-to-Noise Ratio) loss function and the SSIM (Structural Similarity) loss function, and the total loss function It can be expressed as:

[0044]

[0045] Where α1 and α2 are constant coefficients, represents the PSNR loss function, Represents the SSIM loss function.

[0046] In the task of removing rain and fog from images, the design of the loss function should consider both the pixel difference between the generated image and the original image and the similarity of the quality perceived by the human eye. The PSNR loss function It can be expressed as:

[0047]

[0048] Where λ is the loss weight, O represents the output image after the model removes rain and fog, and GT represents the real rain-free and fog-free image. Calculate the mean square error between the predicted value and the target value. ε is a very small constant used to ensure that no numerical errors occur when calculating the logarithm. The PSNR loss function reduces the pixel difference between the generated image and the original image by minimizing the pixel-level mean square error.

[0049] Secondly, the SSIM loss function It can be expressed as:

[0050]

[0051] Where μ represents the average value, σ 2 represents variance, Cov(·) represents covariance, and the SSIM loss function emphasizes the structural similarity of images and evaluates the quality of the derained and fogged image and the original image based on human eye perception.

[0052] Compared with the prior art, the present invention has the following advantages:

[0053] (1) The present invention proposes a method for joint removal of rain and fog from images based on a multi-stage encoder-decoder structure, which combines CNN, which is good at extracting high-frequency features, Swin Transformer, which is good at extracting low-frequency features, and a supervised attention mechanism. These modules recover features from degraded inputs through iterative learning. In addition, the loss function of the present invention is applied to the image recovered at each stage. The method enhances the generalization ability of the model by decomposing the overall recovery process, thereby improving performance.

[0054] (2) In order to better understand the low-frequency features of the image, the present invention proposes an encoder composed of a refocusing convolution module, a dilated convolution module, and a spatial and channel reconstruction convolution module to capture features of different scales. At the same time, a decoder composed of a Swin Transformer and a supervised attention module is used to fuse these low-frequency features.

[0055] (3) The present invention emphasizes the importance of information transfer in effectively achieving the joint removal of rain and fog. To this end, the present invention establishes skip connections between encoders at different stages and introduces a supervised attention module in the decoder to enhance information exchange between stages. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 This is a flow chart of the rain and fog combined removal method proposed by the present invention;

[0057] Figure 2 A network diagram of the image rain and fog joint removal method based on a multi-stage encoder-decoder structure of the present invention;

[0058] Figure 3 This is a rendering of the rain-fog combined removal method of the present invention;

[0059] Figure 4 This is a comparison chart of the effects of the rain and fog combined removal method of the present invention and other image restoration methods. DETAILED DESCRIPTION

[0060] In order to more clearly explain the purpose, technical concept and advantages of the present invention, the technical content of the present invention will be explained in detail and comprehensively based on the attached illustrations. Obviously, what is shown here is only a part of the many embodiments of the present invention, not all. According to the embodiments provided by the present invention, any other embodiments that can be derived by those skilled in the art without performing creative work should also be regarded as within the protection scope of the present invention.

[0061] Figure 1 FIG. 1 is a flowchart of a method for jointly removing rain and fog from an image based on a multi-stage encoder-decoder structure according to an embodiment of the present invention. Figure 1 As shown, the present invention provides a method for jointly removing rain and fog from an image based on a multi-stage encoder-decoder structure, the method comprising the following steps:

[0062] Step 1: Obtain images without rain and fog and images with rain and fog and divide them into training data sets and test data sets. The original images used in the present invention come from two different data sets, namely the Cityscapes data set and the Synscapes data set. The images with rain and fog use the RainCityscapes data set and a new synthetic data set. Based on the Synscapes data set, this data set simulates the foggy effect by simulating atmospheric illumination and transmittance, and uses Gaussian noise and motion blur technology to simulate the rainfall scene. When preparing the data, the data set was divided and randomly divided into 16 groups, 15 of which were used as training data sets, and the remaining 1 group was used as a test data set. Finally, the resolution of all images in the training data set and the test data set synthesized based on the Synscapes data set was adjusted to a uniform 400×400 pixels, and the resolution of all images in the training data set and the test data set in the RainCityscapes data set was adjusted to a uniform 720×360 pixels.

[0063] Figure 2 A schematic diagram of a network for joint rain and fog removal in an image provided by an embodiment of the present invention.

[0064] Step 2: Build a rain and fog joint removal network with a multi-stage encoder-decoder structure. The input of the rain and fog joint removal network is the rain and fog image mentioned in step 1, denoted by I∈R H×W×C , where H represents the height of the image, W represents the width of the image, and C represents the number of channels of the image. Through the 3×3 convolution operation on the input, the shallow features are extracted, which is expressed as F0=H CONV (I). The rain and fog joint removal network with a multi-stage encoder-decoder structure includes five stages of encoders and decoders. The encoder includes three parts: a refocusing convolution module, a hole convolution module, and a space and channel reconstruction convolution module. It aims to extract rich local semantics and high-frequency information. Through the channel attention mechanism, it calibrates, filters, and retains important and representative channel information in the feature map. The encoder feature is represented as:

[0065]

[0066] In the formula Represents the features encoded by the i-th layer encoder. The encoder uses H E (·) indicates that the input of the encoder in the first three stages is the shallow feature F0, and the input of the encoder in the fourth and fifth stages is respectively and

[0067] The refocused convolution module can reparameterize and refocus to enhance the high-frequency feature extraction capability of the network. The formula is as follows:

[0068] F RefConv =H CONV (X⊙H M (X)),

[0069] Where X represents the feature map input, H M (·) is a network for generating the focus matrix, and X⊙H M (X) is a pixel-wise weight redistribution operation. Refocused convolution leverages existing convolution kernels as priors and enhances model performance by learning new representations with trainable refocusing transformations.

[0070] The dilated convolution module can capture multi-scale context information while expanding the receptive field without increasing the number of parameters. The formula is as follows:

[0071]

[0072] Where X represents the feature map input, r represents the expansion rate, and k is the index of the convolution kernel, ranging from k∈[-K, K]. Dilated convolutions provide a feasible solution by expanding the receptive field of the convolution kernel without increasing the number of parameters. This allows the network to better capture contextual information in the image without losing resolution.

[0073] The spatial and channel reconstruction convolution module can reduce the spatial and channel redundancy between features in the network and reduce the computational complexity. The formula is as follows:

[0074] F SCConv =HCR U (H SRU (X)),

[0075] Where H SRU The main focus is on reducing spatial redundancy through a process called spatial segmentation. This process involves segmenting the features along the spatial dimension, performing convolution operations on each segment separately, and then reconstructing the output. CRU Mainly targeting channel redundancy, it effectively reduces the redundant information between channels through channel segmentation and channel reweighting operations.

[0076] F SRU =Concat(H CONV (X i )), i = 1, 2, ...,

[0077] Where X i It means dividing the input X into subsets according to the spatial dimension, then performing spatial parallel convolution on each subset in the spatial dimension, and finally reconstructing the spatial subsets, that is, reconstructing all the convolved subsets back to their original spatial dimensions. Concat means merging subsets.

[0078]

[0079] In the formula It means that the input X is divided into multiple channel dimensions, and then spatially parallel convolution is performed on the subsets in each spatial dimension. Finally, channel subset reconstruction is performed and all the convolved subsets are merged back to the original channel dimension.

[0080] The decoder is composed of a Swin Transformer and a supervised attention module, which aims to extract and aggregate contextual information for global low-frequency features and generate a feature map rich in extensive spatial context and significant deep representation, where the decoder feature is represented as:

[0081]

[0082] In the formula Represents the features decoded by the i-th layer decoder. The decoder uses H D (·)express, It represents the features encoded by the i-th layer encoder, Concat represents the connection operation, the input of the first stage decoder is the features encoded by the first stage encoder, and the input of the second to fifth stages is the features encoded by the encoder of the current stage and the features decoded by the decoder of the previous stage. The two parts are connected along the channel dimension to form a new tensor.

[0083] The SwinTransformer module is used to extract global low-frequency features and aggregate context information. There are three feature tensors in SwinTransformer, namely Query (Q), Key (K) and Value (V). The formula is as follows:

[0084]

[0085] In the formula, Corresponding to a single window and head, M represents the window size, d represents the dimension, and P is the learnable relative position encoding. The model uses modified layer normalization, which differs from standard layer normalization (LayerNorm) in that an additional convolutional layer H is used in the module. RLN (·) to learn the transformation of input mean and standard deviation, thus helping the model capture more complex data distribution characteristics.

[0086]

[0087] Where μ and σ 2 Represent the mean and variance of the input X in the spatial and channel dimensions, respectively. Unlike layer normalization, rectified layer normalization normalizes individual samples in the feature dimension. This makes rectified layer normalization independent of batch size and more suitable for scenarios where batch size changes significantly or sequence length changes dynamically.

[0088] The supervised attention module is able to transfer and transform features between the two stages. The input of the supervised attention module includes the feature map X generated by the Swin Transformer and the original input degraded image containing rain and fog. First, a 1×1 convolution operation transforms X into the spatial dimension of the original input image, changing its number of channels to 3. Then, this transformed feature map is added to the original degraded input image to produce the restored image of this stage. Subsequently, the restored image is processed by a convolution layer with a 1×1 convolution and a sigmoid activation function. This operation generates a pixel-level attention mask, which is the same size as X and indicates the importance of the corresponding pixel. After that, the attention mask is element-wise multiplied with the transformed feature map obtained by the convolution operation. This multiplication aims to calibrate each pixel in the feature map, enhance and emphasize high-value information, while suppressing irrelevant details. Finally, by establishing a residual connection with X, the attention-enhanced feature is obtained, which is the output of the supervised attention module.

[0089] The feature map decoded by the final 5th stage decoder is spatially transformed by the original resolution sub-network and then bitwise added to F0, where the expression is:

[0090]

[0091] Where O represents the output image after the model removes rain and fog, represents the feature map decoded by the 5th stage decoder, H OSR (·) represents the original resolution subnetwork, which is composed of a cascade of multiple original resolution modules for deep processing and enhancing input features. The original resolution module contains multiple channel attention modules and convolutional layers, which are responsible for feature extraction and information fusion.

[0092] Step 3: Use the training data set and loss function in step 1 to train the rain and fog removal network in step 2 until the maximum number of iterations is reached. The loss function includes the PSNR (Peak Signal-to-Noise Ratio) loss function and the SSIM (Structural Similarity) loss function. The total loss function It can be expressed as:

[0093]

[0094] Where α1 and α2 are constant coefficients, represents the PSNR loss function, Represents the SSIM loss function.

[0095] In the task of joint rain and fog removal from images, the design of the loss function should consider both the pixel difference between the generated image and the original image and the similarity of the quality perceived by the human eye. The PSNR loss function can be expressed as:

[0096]

[0097] Where λ is the loss weight, O represents the output image after the model removes rain and fog, and GT represents the real rain-free and fog-free image. Calculate the mean square error between the predicted value and the target value. ε is a very small constant used to ensure that no numerical errors occur when calculating the logarithm. The PSNR loss function reduces the pixel difference between the generated image and the original image by minimizing the pixel-level mean square error.

[0098] Secondly, the SSIM loss function can be expressed as:

[0099]

[0100] Where μ represents the average value, σ 2 represents variance, Cov(·) represents covariance, and the SSIM loss function emphasizes the structural similarity of images and evaluates the quality of the derained and fogged image and the original image based on human eye perception.

[0101] The experiments of the present invention were carried out on two NVIDIARTX 3090 graphics cards, each equipped with 24GB of memory. In terms of software configuration, Ubuntu 18.04 was selected as the operating system, Python as the programming language, and PyTorch1.9 as the deep learning framework, and CUDA 11.2 was used as the compiler. In the training phase, the experiment set 200 training cycles, divided the pictures into patches of 224 sizes, processed 2 samples per batch, and used the AdamW optimizer to adjust and optimize the parameters of the model. The learning rate started at 2e-4, and the weight decay was also set to 2e-4. The experiment used a cosine annealing strategy to adjust the learning rate, gradually reducing it from the initial value to 1e-6.

[0102] Step 4: Input the image to be free of rain and fog into the rain and fog joint removal model obtained by the rain and fog joint removal network trained in step 3 to perform rain and fog joint removal to obtain the final rain and fog removed image.

[0103] Figure 3 This is a diagram showing the effect of jointly removing rain and fog from a rainy and foggy image provided by an embodiment of the present invention.

[0104] like Figure 3As shown in the figure, the results of the combined rain and fog removal of rainy and foggy images synthesized based on the Synscapes dataset are shown. Even in the case of dense fog and heavy rain, due to the extraction and aggregation of multi-stage high-frequency features and low-frequency features, the trained image combined rain and fog removal network can still obtain complex scene images with clear details, clear contours and realism. Specifically, the details of the contours and shapes of objects blocked by dense fog and heavy rain in the image after the combined rain and fog removal show a high degree of realism and naturalness, which further proves the effectiveness of the image combined rain and fog removal method based on a multi-stage encoder-decoder structure proposed in the present invention.

[0105] Figure 4 This is a comparison chart of the effects of the image rain and fog joint removal method proposed in the present invention and other methods on the RainCityscapes data test dataset.

[0106] exist Figure 4 In the figure, the original image containing rain and fog is shown, as well as the images processed by different methods: PCNet, DAF-Net, EPPRNet, Restormer, MB-TaylorFormer and the technology proposed in the present invention. These images also include the real fog-free images corresponding to the rain and fog images. Through comparative analysis, it can be found that MB-TaylorFormer and DAF-Net are more prominent in removing fog and processing low-frequency details, while Restormer, EPPRNet and other methods are more effective in removing high-frequency details and raindrops. However, the method of the present invention has obvious advantages in removing deep fog, distinguishing textures and contours, and processing rain marks in high-frequency information. Therefore, the present technology can effectively remove fog of different densities and various rain marks, thereby improving the clarity of background content and contour structure.

[0107] Finally, it should be noted that: the above describes the specific implementation mode of the present invention with reference to the accompanying drawings. It can be understood that the above is only an embodiment of the present invention and cannot be understood as limiting the scope of protection of the present invention. Changes, modifications, substitutions and variations of the above embodiment made by ordinary technicians in this field within the scope of the present invention should be covered within the scope of protection of the present invention.

Claims

1. A method for joint removal of rain and fog from an image based on a multi-stage encoder-decoder structure, characterized in that: The following steps are involved: Step 1: Obtain images without rain and fog and images with rain and fog, and divide them into training data sets and test data sets. The images without rain and fog can be obtained from a public data set, and the images with rain and fog can be generated by simulating the effects of rain and fog; Step 2: Build a rain and fog joint removal network with a multi-stage encoder-decoder structure. The input of the rain and fog joint removal network is the image with rain and fog in step 1. The image with rain and fog is end-to-end jointly removed from rain and fog, and the clean image after removal is directly output. Step 3: Use the training data set and loss function in step 1 to train the rain and fog removal network in step 2 until the maximum number of iterations is reached; Step 4: For the image that needs to be jointly removed from rain and fog, input it into the rain and fog joint removal network obtained after training in step 3 to obtain the final rain and fog joint removal image.

2. The image rain and fog joint removal method based on a multi-stage encoder-decoder structure according to claim 1 is characterized in that: The rainy and foggy image input in step 2 is denoted as I∈R H×W×C , where H represents the height of the image, W represents the width of the image, and C represents the number of channels of the image. The shallow features of the input I are extracted through a 3×3 convolution operation, which is expressed as F0=H CONV (I) The rain and fog joint removal network of the multi-stage encoder-decoder structure in step 2 includes five stages of encoders and decoders, wherein the encoder of each stage is composed of a refocusing convolution module, a hole convolution module, and a space and channel reconstruction convolution module to extract high-frequency information in the image, and the decoder uses a Swin Transformer module to extract low-frequency information in the image, followed by a supervised attention module to transfer and transform features between the two stages.

3. The method according to claim 2, characterized in that The encoder includes three parts: a refocusing convolution module, a dilated convolution module, and a spatial and channel reconstruction convolution module, which are designed to extract rich local semantics and high-frequency information. Through the channel attention mechanism, the important and representative channel information in the feature map is calibrated, filtered, and retained. The encoder feature is represented as: In the formula Represents the features encoded by the i-th layer encoder. The encoder uses H E (·) indicates that the input of the encoder in the first three stages is the shallow feature F0, and the input of the encoder in the fourth and fifth stages is respectively and The refocused convolution module can reparameterize and refocus to enhance the high-frequency feature extraction capability of the network. The formula is as follows: F RefConv =H CONV (X⊙H M (X)), Where X represents the feature map input, H M (·) is a network for generating the focus matrix, and X⊙H M (X) is a pixel-by-pixel weight redistribution operation. The refocused convolution uses the existing convolution kernel as a prior and enhances the model performance by learning a new representation of the trainable refocusing transformation. The dilated convolution module can capture multi-scale context information while expanding the receptive field without increasing the number of parameters. The formula is as follows: Where X represents the feature map input, r represents the expansion rate, and k is the index of the convolution kernel, ranging from k∈[-K,K]. Dilated convolution provides a feasible solution by expanding the receptive field of the convolution kernel without increasing the number of parameters, which enables the network to better capture the contextual information in the image without losing resolution; The spatial and channel reconstruction convolution module can reduce the spatial and channel redundancy between features in the network and reduce the computational complexity. The formula is as follows: F Scconv =H cRU (H sRU (X)), Where H SRU The main focus is on reducing spatial redundancy through a process called spatial segmentation, which involves segmenting the features along the spatial dimension, performing convolution operations on each segment separately, and then reconstructing the output, H CRU Mainly targeting channel redundancy, it effectively reduces the redundant information between channels through channel segmentation and channel reweighting operations; F SRU =Concat(G CONV (X i )),i=1,2,..., Where X i It means that the input X is divided into subsets according to the spatial dimension, and then spatial parallel convolution is performed on each subset in the spatial dimension. Finally, the spatial subsets are reconstructed, that is, all the convolved subsets are reconstructed back to their original spatial dimensions. Concat means merging subsets. In the formula It means that the input X is divided into multiple channel dimensions, and then spatially parallel convolution is performed on the subsets in each spatial dimension. Finally, channel subset reconstruction is performed and all the convolved subsets are merged back to the original channel dimension.

4. The method according to claim 2, characterized in that: The decoder is composed of a Swin Transformer and a supervised attention module, which aims to extract and aggregate contextual information for global low-frequency features and generate a feature map rich in extensive spatial context and significant deep representation, where the decoder feature is represented as: In the formula Represents the features after decoding by the i-th layer decoder. The decoder uses H D (·)express, Represents the features encoded by the i-th layer encoder, Concat represents the concatenation operation, the input of the first stage decoder is the features encoded by the first stage encoder, and the input of the second to fifth stages is the features encoded by the encoder of the current stage and the features decoded by the decoder of the previous stage. The two parts are concatenated along the channel dimension to form a new tensor; The Swin Transformer module is used to extract global low-frequency features and aggregate context information. There are three feature tensors in Swin Transformer, namely Query (Q), Key (K) and Value (V). The formula is as follows: In the formula, Corresponding to a single window and head, M represents the window size, d represents the dimension, P is the learnable relative position encoding, and the model uses modified layer normalization, which differs from standard layer normalization (LayerNorm) in that an additional convolutional layer H is used in the module. RLN (·) to learn the transformation of input mean and standard deviation, thus helping the model capture more complex data distribution characteristics; Where μ and σ 2 Represent the mean and variance of the input X in the spatial and channel dimensions, respectively. Unlike layer normalization, rectified layer normalization normalizes a single sample in the feature dimension, which makes rectified layer normalization independent of batch size and more suitable for processing scenarios with large changes in batch size or dynamic changes in sequence length; The supervised attention module is capable of transferring and transforming features between the two stages. The input of the supervised attention module includes the feature map X generated by the Swin Transformer and the original input degraded image containing rain and fog. First, a 1×1 convolution operation transforms X into the spatial dimension of the original input image, changing its number of channels to 3. Then, this transformed feature map is added to the original degraded input image to produce the restored image of this stage. Subsequently, the restored image is processed by a convolution layer with a 1×1 convolution and a Sigmoid activation function. This operation generates a pixel-level attention mask, which is the same size as X and indicates the importance of the corresponding pixel. After that, the attention mask is element-wise multiplied with the transformed feature map obtained by the convolution operation. This multiplication aims to calibrate each pixel in the feature map, enhance and emphasize high-value information, and suppress irrelevant details. Finally, by establishing a residual connection with X, the attention-enhanced feature is obtained, which is the output of the supervised attention module.

5. The image rain and fog joint removal method based on a multi-stage encoder-decoder structure according to claim 1 is characterized in that: The clean image after the direct output is removed is decoded by the final 5th stage decoder to obtain a feature map, which is spatially transformed by the original resolution sub-network and then bitwise added to F0, where the expression is: Where O represents the output image after the model removes rain and fog, represents the feature map decoded by the 5th stage decoder, H osR (·) represents the original resolution subnetwork, which is composed of a cascade of multiple original resolution modules for deep processing and enhancing input features. The original resolution module contains multiple channel attention modules and convolutional layers, which are responsible for feature extraction and information fusion.

6. The image rain and fog joint removal method based on a multi-stage encoder-decoder structure according to claim 1 is characterized in that: The loss function in step 3 includes the PSNR (Peak Signal-to-Noise Ratio) loss function and the SSIM (Structural Similarity) loss function. The total loss function It can be expressed as: Where α1 and α2 are constant coefficients, represents the PSNR loss function, represents the SSIM loss function; In the task of removing rain and fog from images, the design of the loss function should consider both the pixel difference between the generated image and the original image and the similarity of the quality perceived by the human eye. The PSNR loss function It can be expressed as: Where λ is the loss weight, O represents the output image after the model removes rain and fog, and GT represents the real rain-free and fog-free image. Calculate the mean square error between the predicted value and the target value. ε is a very small constant used to ensure that no numerical errors occur when calculating the logarithm. The PSNR loss function reduces the pixel difference between the generated image and the original image by minimizing the pixel-level mean square error. Secondly, the SSIM loss function It can be expressed as: Where μ represents the average value, σ 2 represents variance, Cov(·) represents covariance, and the SSIM loss function emphasizes the structural similarity of images and evaluates the quality of the derained and fogged image and the original image based on human eye perception.

Images