Method and system for removing multiple severe weather conditions from images

Abstract

The present invention discloses a method and system for removing multiple severe weather events from an image. The method comprises the following steps: obtaining an image to be restored, including a degraded image affected by multiple concurrent severe weather events; downsampling the image to be restored; constructing a multiple severe weather removal model, inputting the downsampled image to be restored into the multiple severe weather removal model, which removes multiple concurrent severe weather events from the degraded image to be restored, generating a restored degraded image; collaboratively training a backbone network and auxiliary branches based on a multiple degradation robust loss to obtain a trained multiple severe weather removal model; and obtaining a restored image of the degraded image to be restored based on the trained multiple severe weather removal model. The present invention constructs a multiple concurrent severe weather removal paradigm for a downsampling-removal-reconstruction process, significantly reducing the difficulty of removing multiple concurrent severe weather events and improving the quality of image restoration.

Description

Technical Field

[0001] The present invention relates to the technical field of image restoration, and in particular to a method and system for removing multiple severe weather conditions from an image. Background Art

[0002] Image inpainting is a low-level vision task encompassing deblurring, denoising, and super-resolution. These tasks aim to restore or reconstruct high-quality versions from low-quality images, thereby improving human visual perception and the performance of many outdoor vision systems. The imaging quality of scenes in outdoor environments is inevitably affected by various adverse weather conditions (e.g., rain, fog, snow, and raindrops). Therefore, removing degradation caused by adverse weather conditions has gradually become a specialized image inpainting task and has attracted widespread attention in computer vision research. Recently, several studies have demonstrated significant results in removing adverse weather conditions, but they all target a single type of adverse weather and are difficult to transfer to other types. To address this issue, some methods have attempted to develop a framework to remove multiple degradations, but these methods typically require specific model training for different degradations, i.e., training a set of model parameters for each degradation, which is inefficient and limited in practical applications.

[0003] In bad weather removal and other restoration tasks such as deblurring and denoising, each image degradation (e.g., bad weather, blur, noise, etc.) presents its own unique perturbation pattern that needs to be processed or restored. Moreover, in the real world, removing bad weather from outdoor images affected by only a single type of degradation is overly idealistic, as multiple bad weather conditions often coexist in the same scene. However, due to the significant differences in feature distributions between different degradation types, the above methods often perform poorly in this case. This is because modeling the relationship between various degradation forms becomes particularly challenging when dealing with concurrent and complex degradation scenarios. Summary of the Invention

[0004] In order to overcome the defects and shortcomings of the existing technology, the present invention provides a method and system for removing multiple severe weather events in images. The present invention constructs a multiple concurrent severe weather removal paradigm with a downsampling-removal-reconstruction process, and applies downsampling technology to the removal of multiple concurrent severe weather events, thereby compressing the component particles of different severe weather events and reducing the absolute disturbance caused by degradation.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] The present invention provides a method for removing multiple severe weather conditions from an image, comprising the following steps:

[0007] Acquire images to be restored, including degraded images affected by multiple concurrent severe weather events;

[0008] Downsample the image to be repaired;

[0009] Constructing a multiple severe weather removal model, inputting the downsampled image to be restored into the multiple severe weather removal model, wherein the multiple severe weather removal model removes multiple concurrent severe weather events in the degraded image to be restored, thereby generating a restored degraded image;

[0010] The multiple severe weather removal model includes a backbone network, auxiliary branches and multiple degradation robust losses;

[0011] The backbone network includes an inherent shallow feature extraction block, an inherent deep feature extraction block, a first inherent feature fusion block, and a first high-resolution image reconstruction module. The inherent shallow feature extraction block is used to extract inherent shallow features of the downsampled image to be repaired, the inherent deep feature extraction block is used to extract inherent deep features of the inherent shallow features, the first inherent feature fusion block is used to interact and fuse the inherent shallow features and the inherent deep features across channels to generate an enhanced feature representation, and the first high-resolution image reconstruction module is used to extract effective feature information under the supervision of high-resolution true values ​​and reconstruct a repaired image corresponding to the degraded image to be repaired;

[0012] The auxiliary branch includes a second intrinsic feature fusion block and a second high-resolution image reconstruction module, wherein the second intrinsic feature fusion block is used to interact and fuse the intrinsic shallow features and the intrinsic deep features across channels, and the second high-resolution image reconstruction module is used to reconstruct the degraded output under the supervision of the high-resolution degraded input image and output the high-resolution degraded image;

[0013] Based on multiple degradation robust losses, the backbone network and auxiliary branches are trained collaboratively to obtain a trained multiple severe weather removal model.

[0014] The degraded image to be repaired is obtained based on the trained multiple severe weather removal model to obtain the repaired image.

[0015] As a preferred technical solution, the inherent shallow feature extraction block uses a convolutional layer to extract the inherent shallow feature f S , specifically expressed as:

[0016] f S =IFE S (I′ l ↓)=Conv(I′ l ↓)

[0017] Among them, I′ l ↓ indicates the image to be repaired after downsampling, IFE S(.) represents the inherent shallow feature extraction block, and Conv(.) represents the convolutional layer, which maps the downsampled image space to be repaired to a higher-dimensional feature space.

[0018] As a preferred technical solution, the inherent deep feature extraction block includes several residual Swin Transformers and a convolutional layer. Each residual Swin Transformer includes several Swin Transformer layers and a convolutional layer. The output of the nth Swin Transformer layer is expressed as:

[0019] f m,n =STL m,n (f m,n-1 )

[0020] Among them, f m,n Represents the intermediate features obtained from the nth Swin Transformer layer, STL m,n (·) represents the nth Swin Transformer layer in the mth residual Swin Transformer, f m,n-1 represents the intermediate features obtained from the n-1th SwinTransformer layer;

[0021] A convolutional layer is added before the residual connection, and the residual Swin Transformer is expressed as:

[0022] f m =RSTB m (f m-1 )=Conv(f m,N )+f m,0

[0023] Among them, f m represents the output of the mth residual Swin Transformer, f m-1 represents the output of the m-1th residual SwinTransformer, f m,N represents the output of the Nth Swin Transformer layer in the mth residual Swin Transformer, f m,0 Represents the input of the mth residual Swin Transformer. The inherent shallow features are input into the residual Swin Transformer to extract the inherent deep features. RSTB m represents the mth residual Swin Transformer, Conv represents convolution;

[0024] The inherent deep features are obtained by passing through several Swin Transformer layers and finally a convolutional layer.

[0025] As a preferred technical solution, the feature input obtained by the previous Swin Transformer layer is represented by a two-dimensional feature map X, which is divided into non-overlapping windows of size μ×μ. Local self-attention is calculated for each feature in each window, and relative position encoding is applied to the attention module, which is specifically expressed as:

[0026]

[0027] Q=X i P Q

[0028] K=X i P K

[0029] V=X i P V

[0030] Among them, P Q 、P K 、 is the projection matrix shared across different windows, i is the index of the feature of the i-th local window, B represents the relative position deviation containing the learnable parameters, and C represents the number of channels;

[0031] For a given local window feature from the n-1th Swin Transformer layer The calculation of the current SwinTransformer layer is expressed as:

[0032]

[0033] Among them, LN represents layer normalization, MSA and MLP represent multi-head self-attention and multi-layer perceptron respectively. and Respectively represent the output of the multi-layer perceptron and multi-head self-attention after residual connection;

[0034] Add a normalization layer before the multi-layer perceptron and multi-head self-attention.

[0035] As a preferred technical solution, the first inherent feature fusion block is based on a convolutional layer with LeakyReLU, and uses residual connections to complete the channel interaction and fusion of inherent shallow features and inherent deep features to generate a new feature representation f out , specifically expressed as:

[0036] f out =IFF(fS +f D )=LR(Conv(f S +f p ))

[0037] Among them, IFF(·) represents the first intrinsic feature fusion block, LR(·) represents the LeakyReLU activation function, Conv represents convolution, and f S represents the inherent shallow features, f D Indicates inherent deep features.

[0038] As a preferred technical solution, the first high-resolution image reconstruction module obtains the enhanced feature representation generated by the first intrinsic feature fusion block, learns a set of upsampling filters based on the sub-pixel convolution layer, and reconstructs the target image under the supervision of the high-resolution true value, which is specifically expressed as follows:

[0039]

[0040] Where PS(·( represents the pixel shuffling operation, HRIR(·) represents the first high-resolution image reconstruction module, and f out It represents the enhanced feature representation generated by the first intrinsic feature fusion block, and Conv represents convolution.

[0041] As a preferred technical solution, based on multiple degradation robust losses, the backbone network and auxiliary branches are collaboratively trained to obtain a trained multiple severe weather removal model, specifically including:

[0042] Obtaining a degraded image sample to be repaired and a clean image sample corresponding to the degraded image sample to be repaired;

[0043] After downsampling the degraded image samples to be repaired, they are input into the backbone network to extract inherent shallow features and inherent deep features. The inherent shallow and deep features are interacted and fused to generate enhanced feature representations, and the repaired image corresponding to the degraded image to be repaired is reconstructed;

[0044] The inherent shallow and deep features are interacted and fused to reconstruct the degraded output and output a high-resolution degraded image;

[0045] Introducing multiple degradation robust losses to constrain the feature extraction process through inherent shallow feature extraction blocks with shared weights;

[0046] The predicted restored image is compared with the corresponding clean image sample until the multiple severe weather removal model is determined to have converged, and the iterative training process ends.

[0047] As a preferred technical solution, in the collaborative training process of the backbone network and the auxiliary branches, select N d For degraded images and the repaired image Form a batch of positive samples, denoted as

[0048] The positive samples are input into the batch inherent shallow feature extraction block to obtain the corresponding shallow feature set;

[0049] Using an additional 2(N d -1 samples form a negative sample batch and input the inherent shallow feature extraction block to obtain the corresponding shallow feature set.

[0050] As a preferred technical solution, in the collaborative training process of the backbone network and the auxiliary branches, the overall loss function includes: pixel loss function, perceptual loss function, negative structural similarity loss function and multiple degradation robust loss function, which can be specifically expressed as:

[0051]

[0052] in, is the pixel loss function, I′ l is the image sample to be repaired, is the reconstructed degraded image sample corresponding to the image sample to be repaired, I l is the clean image sample corresponding to the image sample to be repaired, is the restored image sample corresponding to the image sample to be restored, represents element-wise subtraction, represents the mean operation, is the perceptual loss function, φ q represents the operation of extracting features from the input image, is the negative structural similarity loss function, SSIM represents the structural similarity calculation function, is the multiple degradation robust loss function, Refers to the feature set extracted from other degraded images and their corresponding real images in the same batch, N d Indicates the number of sample pairs, sim cos Represents the cosine similarity between two features.

[0053] The present invention also provides a system for removing multiple severe weather conditions from images, which is used to implement the above-mentioned method for removing multiple severe weather conditions from images. The system includes: an image acquisition module to be repaired, a downsampling module, a model building module, a collaborative training module, and a repaired image output module;

[0054] The image acquisition module to be restored is used to acquire images to be restored, including degraded images affected by multiple concurrent severe weather conditions;

[0055] The downsampling module is used to downsample the image to be repaired;

[0056] The model building module is used to build a multiple severe weather removal model, input the downsampled image to be repaired into the multiple severe weather removal model, and the multiple severe weather removal model removes multiple concurrent severe weather events in the degraded image to be repaired to generate a repaired degraded image;

[0057] The multiple severe weather removal model includes a backbone network, auxiliary branches and multiple degradation robust losses;

[0058] The backbone network includes an inherent shallow feature extraction block, an inherent deep feature extraction block, a first inherent feature fusion block, and a first high-resolution image reconstruction module. The inherent shallow feature extraction block is used to extract inherent shallow features of the downsampled image to be repaired, the inherent deep feature extraction block is used to extract inherent deep features of the inherent shallow features, the first inherent feature fusion block is used to interact and fuse the inherent shallow features and the inherent deep features across channels to generate an enhanced feature representation, and the first high-resolution image reconstruction module is used to extract effective feature information under the supervision of high-resolution true values ​​and reconstruct a repaired image corresponding to the degraded image to be repaired;

[0059] The auxiliary branch includes a second intrinsic feature fusion block and a second high-resolution image reconstruction module, wherein the second intrinsic feature fusion block is used to interact and fuse the intrinsic shallow features and the intrinsic deep features across channels, and the second high-resolution image reconstruction module is used to reconstruct the degraded output under the supervision of the high-resolution degraded input image and output the high-resolution degraded image;

[0060] The collaborative training module is used to collaboratively train the backbone network and the auxiliary branches based on multiple degradation robust losses to obtain a trained multiple severe weather removal model;

[0061] The repaired image output module is used to input the degraded image to be repaired into the trained multiple severe weather removal model to obtain a repaired image.

[0062] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0063] (1) The present invention constructs a downsampling-removal-reconstruction process, which compresses the constituent particles of severe weather and reduces the absolute disturbance caused by various degradation phenomena, greatly reducing the difficulty of removing multiple concurrent severe weather events and improving the quality of image restoration. The multiple severe weather removal model can bring a powerful performance improvement to the removal processing of multiple concurrent severe weather events in images.

[0064] (2) The present invention downsamples the degraded image affected by severe weather using a bicubic interpolation algorithm to obtain a low-resolution version thereof and inputs the low-resolution version into the backbone network. The low-resolution version is processed by an inherent shallow feature extraction block and an inherent deep feature extraction block to extract inherent features. In order to better guide feature learning, a first inherent feature fusion block is used to interact and fuse the inherent shallow and deep features according to their channels to generate a new feature representation. The first high-resolution image reconstruction module uses the above feature output to generate a repaired degraded image corresponding to the degraded image to be repaired. At the same time, an auxiliary branch equipped with a second inherent feature fusion block and a second high-resolution image reconstruction module optimizes the inherent features through fine-grained structural representation under the supervision of the high-resolution degraded input, thereby guiding and optimizing the learning of feature representation.

[0065] (3) In order to ensure robustness to multiple concurrent degradation phenomena, a multiple degradation robust loss is introduced to better constrain the feature extraction process through inherent shallow feature extraction blocks with shared weights, forming collaborative training. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 Schematic diagram of the process of the method for removing multiple severe weather conditions from an image according to the present invention;

[0067] Figure 2 Schematic diagram of the network architecture of the multiple severe weather removal model of the present invention;

[0068] Figure 3 (a) is a schematic diagram of the network architecture of the residual Swin Transformer of the present invention;

[0069] Figure 3 (b) is a schematic diagram of the network architecture of the Swin Transformer layer of the present invention. DETAILED DESCRIPTION

[0070] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0071] Example 1

[0072] like Figure 1 As shown, this embodiment provides a method for removing multiple severe weather conditions from an image, comprising the following steps:

[0073] S1: Acquire an image to be restored, wherein the image to be restored includes a degraded image affected by multiple concurrent severe weather conditions;

[0074] S2: downsampling the image to be repaired;

[0075] Specifically, the image to be repaired is downsampled using a bicubic interpolation algorithm to obtain its low-resolution version;

[0076] S3: constructing a multiple bad weather removal model, inputting the image to be repaired into the multiple bad weather removal model, so that the multiple bad weather removal model removes multiple concurrent bad weather in the degraded image to be repaired, and generates a clean repaired degraded image;

[0077] like Figure 2 As shown in the figure, the multiple bad weather removal model includes a backbone network, auxiliary branches and multiple degradation robust losses. The backbone network is used to extract the inherent features of the degraded image, interact and fuse the inherent shallow and deep features, and generate a repaired clean image corresponding to the degraded image to be repaired through the generated new feature representation.

[0078] Specifically, the backbone network includes an inherent shallow feature extraction block, an inherent deep feature extraction block, a first inherent feature fusion block and a first high-resolution image reconstruction module. The inherent shallow feature extraction block and the inherent deep feature extraction block are used to process the downsampled degraded input and convert it into features with semantic meaning. The first inherent feature fusion block is used for inter-channel interaction and fusion of features to generate enhanced feature representation. Under the supervision of high-resolution true values, the first high-resolution image reconstruction module uses the extracted effective feature information to reconstruct the repaired clean image corresponding to the degraded image to be repaired.

[0079] In this embodiment, the intrinsic shallow feature extraction block applies a simple convolutional layer to extract the intrinsic shallow features f S , the formula is as follows:

[0080] f S =IFE S (I l ′ ↓)=Conv(I l ′ ↓)

[0081] Among them, I′ l ↓ represents the degraded input after downsampling, IFE S (.) is an inherent shallow feature extraction block, specifically a convolutional layer, which can map the input image space to a higher-dimensional feature space, thereby improving the optimization stability without increasing unnecessary computational costs, and is more consistent with the design of the model in this embodiment.

[0082] Since the residual Swin Transformer block performs well in capturing long-distance dependencies. In the solution of this embodiment, thanks to downsampling, the application of the residual Swin Transformer method based on local windows on low-resolution feature maps can be processed on a larger receptive field, enabling the network to better learn long-distance dependencies. Based on these considerations, the residual Swin Transformer will be used as a block suitable for the solution of this embodiment to perform local attention and cross-window interaction, thereby constructing an inherent deep feature extraction block. The inherent shallow features are further extracted by the inherent deep feature extraction block to obtain the inherent deep features f D , the formula is as follows:

[0083] f D =IFE D (f S )

[0084] Among them, IFE D (·) is the inherent deep feature extraction block, which consists of several residual Swin Transformers and a convolutional layer; Figure 3 (a) Figure 3 As shown in (b), the residual Swin Transformer is a residual block that integrates several Swin Transformer layers and a convolutional layer. In the mth residual Swin Transformer, given the input feature f m,0 , the output of the nth Swin Transformer layer is expressed as:

[0085] f m,n =STL m,n (f m,n-1 ),n∈{1,2,…,N}

[0086] Among them, f m,n Represents the intermediate features obtained from the nth Swin Transformer layer, STL m,n (·) is the nth Swin Transformer layer in the mth residual Swin Transformer, f m,n-1 represents the intermediate features obtained from the n-1th SwinTransformer layer, and N represents the number of Swin Transformer layers. In order to enhance translation equivariance and achieve aggregation of features at different levels, a convolutional layer is added before the residual connection. Mathematically, the residual SwinTransformer can be defined as:

[0087] f m =RSTB m (f m-1)=Conv(f m,N )+f m,0 ,m∈{1,2,…,M}

[0088] Among them, f m is the output of the mth residual Swin Transformer, f m-1 represents the output of the m-1th residual SwinTransformer, f m,N represents the output of the Nth Swin Transformer layer in the mth residual Swin Transformer, f m,0 Represents the input of the mth residual Swin Transformer, the inherent shallow feature f S The input residual SwinTransformer is used for further intrinsic deep feature extraction. The intrinsic deep features obtained in this way contain high-frequency information, which is crucial for pixel recovery and detail preservation. M represents the number of residual Swin Transformers, RSTB m Represents the mth residual Swin Transformer.

[0089] In particular, the Swin Transformer layer performs self-attention within non-overlapping local windows. Given a 2D feature map X, i.e., the feature input obtained from the previous Swin Transformer layer, represented by a 2D feature map X, X is divided into non-overlapping windows of size μ×μ, and features are then extracted from each window i. Where C represents the number of channels. Next, the Swin Transformer layer performs local self-attention calculations on each feature within each window and applies relative position encoding to the attention module. i The calculation formula of the attention mechanism is:

[0090] Q=X i P Q , K=X i P K , V=X i P V ,

[0091]

[0092] Among them, P Q 、P K 、 is the projection matrix shared across different windows, where C represents the number of channels. i∈{1,2,…,I} is the index of the i-th local window feature. B represents the relative position bias containing the learnable parameters. The Swin Transformer layer in the residual Swin Transformer includes two core components: a multi-layer perceptron and a multi-head self-attention, as shown in Figure 3 (a) Figure 3 (b) shows that for a given local window feature from the n-1th Swin Transformer layer The calculation of the current Swin Transformer layer is expressed as:

[0093]

[0094]

[0095] Among them, LN represents layer normalization, MSA and MLP represent multi-head self-attention and multi-layer perceptron respectively. and They represent the outputs of the multi-layer perceptron and multi-head self-attention after residual connections. A layer normalization layer is added before the multi-layer perceptron and multi-head self-attention. The Swin Transformer layer performs self-attention in a local window, which significantly reduces the computational complexity compared to global self-attention. The inherent deep features f are obtained through several Swin Transformer layers and finally a convolutional layer. D .

[0096] In this embodiment, the first intrinsic feature fusion block more effectively mines image reconstruction information and enhances the modeling capability of latent feature distribution. It uses convolutional layers with LeakyReLU and adopts residual connections to promote cross-channel interaction and fusion of shallow and deep features to generate a new feature representation f out , the formula is as follows:

[0097] f out =IFF(f S +f D )=LR(Conv(f S +f D ))

[0098] Among them, IFF(·) represents the first intrinsic feature fusion block, LR(·) is the LeakyReLU activation function, through a long skip connection, the shallow feature f S Directly input into the first intrinsic feature fusion block to preserve low-frequency information.

[0099] In this embodiment, the first high-resolution image reconstruction module transforms the final feature f obtained from the first intrinsic feature fusion block into out , using a sub-pixel convolutional layer that learns a set of upsampling filters to upsample the final low-resolution feature map to a high-resolution clean output Under the supervision of high-resolution true values, the first high-resolution image reconstruction module reconstructs clean targets and promotes the recovery of more details. The process can be summarized as follows:

[0100]

[0101] Here, PS(·) represents the pixel shuffling operation, and HRIR(·) denotes the first high-resolution image reconstruction module. In this module, two convolutional layers are designed at the beginning and end. The former increases the number of channels, providing sufficient data for the subsequent pixel shuffling and enabling efficient upsampling. The latter adjusts and refines the feature maps after upsampling to ensure that the output image has the correct number of channels.

[0102] Combine Figure 2 As shown, the auxiliary branch includes a second intrinsic feature fusion block and a second high-resolution image reconstruction module, which is used to optimize the intrinsic features through fine-grained structure representation, aiming to guide and optimize the learning of feature representation;

[0103] Specifically, by employing downsampling techniques to process images under various adverse weather conditions, the compositional particles present in these conditions are effectively mitigated from a low-resolution perspective, and the absolute perturbations caused by degradation are reduced. While the backbone network is capable of reconstructing high-resolution clean outputs from low-resolution degraded inputs, an auxiliary branch is designed to enhance its ability to capture local contextual information, which enables more refined reconstruction of high-resolution details and overcomes the challenges posed by limited effective image information. The auxiliary branch comprises a second intrinsic feature fusion block and a second high-resolution image reconstruction module, which do not share weights with their counterparts in the backbone network.

[0104] The input of the auxiliary branch is the result of concat- ing the inherent shallow features and the inherent deep features. The second inherent feature fusion block utilizes a convolutional layer with LeakyReLU and adopts residual connections to promote cross-channel interaction and fusion of inherent shallow features and inherent deep features. The second high-resolution image reconstruction module reconstructs the degraded output under the supervision of the high-resolution degraded input image. The output is a reconstructed high-resolution degraded image, thereby effectively realizing the reconstruction of fine-grained structural information of the image under low-resolution input.

[0105] By guiding and optimizing feature representation learning through auxiliary branches, the model overcomes limitations in capturing internal pixel relationships. As a result, it effectively eliminates multiple degradations while preserving reconstruction quality, demonstrating attention to detail and accurate modeling even with only low-resolution severe weather imagery. To avoid additional computational cost, the auxiliary branches are removed during inference.

[0106] S4: Based on the multiple degradation robust loss, the backbone network and auxiliary branches are trained together to obtain the trained multiple severe weather removal model;

[0107] In this embodiment, collaborative training is performed by introducing contrast constraints between the backbone network and the auxiliary branches, and multiple degradation robust losses are used to better constrain the feature extraction process through inherent shallow feature extraction blocks with shared weights, forming collaborative training.

[0108] In this embodiment, the backbone network is trained with clean labeled data, while the auxiliary branches are trained under supervision with degraded original (un-downsampled) input. For concurrent multiple severe weather images, it is necessary to ensure that areas not covered by severe weather can be well restored during the training process. The auxiliary branches are added to ensure that the details of these uncovered areas remain consistent. In addition, the effective pixels of the image are limited, so that the limited pixel information can be maximized.

[0109] In this embodiment, given that there are significant differences in the feature distributions of multiple severe weather elements in the same scene, a multiple degradation robust loss is introduced to promote the learned representation to be robust to multiple concurrent degradation phenomena. By promoting the collaborative training of the backbone network and the auxiliary branches, the outputs of the backbone network and the auxiliary branches share common properties and the embedding in the latent space should also have a low distance. Therefore, contrast constraints are imposed in the shallow feature space, and appropriate positive and negative pairs are constructed to further take advantage of the learned related feature distributions to obtain sufficient information to reconstruct a clean background.

[0110] Specifically, the outputs of the auxiliary branches and the backbone network are mapped to the embedding space through the inherent shallow feature extraction block with shared weights, and the similarity metric is optimized to make them closer. It is expected that the refined features can resist multiple degradations in bad weather images. Specifically, during the training process, N is first selected. d For degraded output and clean output Form a batch of positive samples, denoted as These images are fed into the intrinsic shallow feature extraction block to obtain the shallow feature set To constrain the common features in the embedding, an additional 2(N d-1) samples form a negative batch and feed them into the intrinsic shallow feature extraction block. By encouraging positive samples to be closer to and maintain a large distance from negative samples, a multi-degradation robust loss constraint is imposed, thereby establishing associations between representations with similar feature distributions and contrasting them with dissimilar ones. The ultimate goal is to promote collaboration among different modules and help the intrinsic shallow feature extraction block accurately extract information relevant to image reconstruction.

[0111] In this embodiment, the training process of the multiple severe weather removal model includes:

[0112] Repeat the following training operations for the multiple severe weather removal model until the multiple severe weather removal model is determined to have converged:

[0113] Obtaining a sample of a degraded image to be repaired and a clean image sample corresponding to the sample degraded image to be repaired;

[0114] Before inputting the degraded image sample to be repaired into the backbone network, it is first downsampled to obtain a low-resolution version. The low-resolution degraded image is then processed by the intrinsic shallow feature extraction block and the intrinsic deep feature extraction block to extract intrinsic features. The first intrinsic feature fusion block then interacts and fuses the intrinsic shallow and deep features according to their channels to generate a new feature representation. The first high-resolution image reconstruction module then uses these feature outputs to reconstruct the repaired clean image corresponding to the degraded image to be repaired. The second intrinsic feature fusion block interacts and fuses the intrinsic shallow and deep features across channels. The second high-resolution image reconstruction module reconstructs the degraded output under the supervision of the high-resolution degraded input image. This output is a reconstructed high-resolution degraded image, effectively reconstructing fine-grained structural information from the image given the low-resolution input. Furthermore, a multi-degradation robust loss is introduced to constrain the feature extraction process through the intrinsic shallow feature extraction blocks with shared weights.

[0115] The predicted restored clean image is compared with the corresponding clean image sample to compare the image prediction accuracy and the image restoration effect. When it is determined based on the comparison results that the multiple severe weather removal model has not converged, the updated sample clean image to be restored and the sample clean image corresponding to the sample degraded image to be restored are obtained.

[0116] In this embodiment, when training the multiple severe weather removal models, the following steps are also included:

[0117] The multiple severe weather removal models are updated according to their overall loss function, wherein the overall loss function is composed of a pixel loss function, a perceptual loss function, a negative structural similarity loss function, and a multiple degradation robustness loss function;

[0118] The pixel loss function is calculated according to the following formula:

[0119]

[0120] in, is the pixel loss function, I′ l is the image sample to be repaired, is the reconstructed degraded image sample corresponding to the image sample to be repaired, I l is the clean image sample corresponding to the image sample to be repaired, is the sample repaired image corresponding to the sample image to be repaired, represents element-wise subtraction, represents the mean operation;

[0121] The perceptual loss function is calculated according to the following formula:

[0122]

[0123] in, is the perceptual loss function, φ q It represents the operation of extracting features from the input image, specifically the output feature map of the qth layer of a pre-trained convolutional neural network (such as VGG, ResNet, etc.). Its function is to map the image into a high-dimensional feature space and calculate the perceptual loss by comparing the differences between these feature maps. In this embodiment, the conv2,3 layers of the VGG-16 network pre-trained on ImageNet are preferably used to extract features, and the distance between these features is calculated as a measure of perceptual similarity.

[0124] The negative structural similarity loss function is calculated according to the following formula:

[0125]

[0126] in, is the negative structural similarity loss function;

[0127] The multiple degradation robust loss function is calculated according to the following formula:

[0128]

[0129] in, is the multiple degradation robust loss function, Refers to the feature set extracted from other degraded images and their corresponding real images in the same batch, sim cos represents the cosine similarity between two features, and the temperature parameter τ is set to 0.25;

[0130] The overall loss function of the multiple severe weather removal model is calculated according to the following formula:

[0131]

[0132] in, is the overall loss function, λ MDR The first preset parameter, λ p is the second preset parameter.

[0133] S5: The degraded image to be repaired is obtained based on the trained multiple bad weather removal model to obtain the repaired image.

[0134] The present invention allows for custom downsampling ratios, achieving a flexible balance between memory requirements and restoration performance. Furthermore, the model employs upsampling (i.e., pixel shuffling) in both the first high-resolution image reconstruction module in the backbone network and the second high-resolution image reconstruction module in the auxiliary branch. This allows the model to not only eliminate the effects of inclement weather on low-resolution input but also reconstruct the corresponding high-resolution image. This feature makes the model highly adaptable to various practical applications requiring high-resolution images, something that previous methods could not achieve.

[0135] Example 2

[0136] This embodiment provides a system for removing multiple severe weather conditions from an image, which is used to implement the method for removing multiple severe weather conditions from an image in the first embodiment. The system includes: an image acquisition module to be repaired, a downsampling module, a model building module, a collaborative training module, and a repaired image output module.

[0137] In this embodiment, the module for acquiring images to be restored is used to acquire images to be restored, including degraded images affected by multiple concurrent severe weather conditions;

[0138] In this embodiment, the downsampling module is used to downsample the image to be repaired;

[0139] In this embodiment, the model building module is used to build a multiple severe weather removal model. The downsampled image to be repaired is input into the multiple severe weather removal model. The multiple severe weather removal model removes multiple concurrent severe weather events in the degraded image to be repaired to generate a repaired degraded image.

[0140] In this embodiment, the multiple severe weather removal model includes a backbone network, auxiliary branches, and multiple degradation robust losses;

[0141] In this embodiment, the backbone network includes an inherent shallow feature extraction block, an inherent deep feature extraction block, a first inherent feature fusion block, and a first high-resolution image reconstruction module. The inherent shallow feature extraction block is used to extract inherent shallow features of the downsampled image to be repaired, the inherent deep feature extraction block is used to extract inherent deep features of the inherent shallow features, the first inherent feature fusion block is used to interact and fuse the inherent shallow features and the inherent deep features across channels to generate an enhanced feature representation, and the first high-resolution image reconstruction module is used to extract effective feature information under the supervision of high-resolution true values ​​and reconstruct a repaired image corresponding to the degraded image to be repaired;

[0142] In this embodiment, the auxiliary branch includes a second intrinsic feature fusion block and a second high-resolution image reconstruction module, wherein the second intrinsic feature fusion block is used to interact and fuse the intrinsic shallow features and the intrinsic deep features across channels, and the second high-resolution image reconstruction module is used to reconstruct the degraded output under the supervision of the high-resolution degraded input image and output a high-resolution degraded image;

[0143] In this embodiment, the collaborative training module is used to collaboratively train the backbone network and the auxiliary branches based on multiple degradation robust losses to obtain a trained multiple severe weather removal model;

[0144] In this embodiment, the restored image output module is used to input the degraded image to be restored into the trained multiple severe weather removal model to obtain a restored image.

[0145] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be considered as equivalent replacement methods and are included in the scope of protection of the present invention.

Claims

1. A method for removing multiple bad weather conditions from an image, characterized in that: The steps include: Acquire images to be restored, including degraded images affected by multiple concurrent severe weather events; Downsample the image to be repaired; Constructing a multiple severe weather removal model, inputting the downsampled image to be restored into the multiple severe weather removal model, wherein the multiple severe weather removal model removes multiple concurrent severe weather events in the degraded image to be restored, thereby generating a restored degraded image; The multiple severe weather removal model includes a backbone network, auxiliary branches and multiple degradation robust losses; The backbone network includes an inherent shallow feature extraction block, an inherent deep feature extraction block, a first inherent feature fusion block, and a first high-resolution image reconstruction module. The inherent shallow feature extraction block is used to extract inherent shallow features of the downsampled image to be repaired, the inherent deep feature extraction block is used to extract inherent deep features of the inherent shallow features, the first inherent feature fusion block is used to interact and fuse the inherent shallow features and the inherent deep features across channels to generate an enhanced feature representation, and the first high-resolution image reconstruction module is used to extract effective feature information under the supervision of high-resolution true values ​​and reconstruct a repaired image corresponding to the degraded image to be repaired; The auxiliary branch includes a second intrinsic feature fusion block and a second high-resolution image reconstruction module, wherein the second intrinsic feature fusion block is used to interact and fuse the intrinsic shallow features and the intrinsic deep features across channels, and the second high-resolution image reconstruction module is used to reconstruct the degraded output under the supervision of the high-resolution degraded input image and output the high-resolution degraded image; Based on multiple degradation robust losses, the backbone network and auxiliary branches are trained collaboratively to obtain a trained multiple severe weather removal model. The degraded image to be repaired is obtained based on the trained multiple severe weather removal model to obtain the repaired image.

2. The method for removing multiple severe weather conditions from an image according to claim 1, wherein: The inherent shallow feature extraction block uses a convolutional layer to extract the inherent shallow features f S , specifically expressed as: f S =IFE S (I′ l ↓)=conv(I′ l ↓) Among them, I′ l ↓ indicates the image to be repaired after downsampling, IFE S (.) represents the inherent shallow feature extraction block, and conv(.) represents the convolutional layer, which maps the downsampled image space to be repaired to a higher-dimensional feature space.

3. The method for removing multiple severe weather conditions from an image according to claim 1, wherein: The inherent deep feature extraction block includes several residual Swin Transformers and a convolutional layer. Each residual Swin Transformer includes several Swin Transformer layers and a convolutional layer. The output of the nth Swin Transformer layer is expressed as: f m,n =STL m,n (f m,n-1 ) Among them, f m,n Represents the intermediate features obtained from the nth Swin Transformer layer, STL m,n (·) represents the nth Swin Transformer layer in the mth residual Swin Transformer, f m,n-1 represents the intermediate features obtained from the n-1th SwinTransformer layer; A convolutional layer is added before the residual connection, and the residual Swin Transformer is expressed as: f m =RSTB m (f m-1 )=Conv(f m,N )+f m,0 Among them, f m represents the output of the mth residual Swin Transformer, f m-1 represents the output of the m-1th residual SwinTransformer, f m,N represents the output of the Nth Swin Transformer layer in the mth residual Swin Transformer, f m,0 Represents the input of the mth residual Swin Transformer. The inherent shallow features are input into the residual Swin Transformer to extract the inherent deep features. RSTB m represents the mth residual Swin Transformer, Conv represents convolution; The inherent deep features are obtained by passing through several Swin Transformer layers and finally a convolutional layer.

4. The method for removing multiple severe weather conditions from an image according to claim 3, wherein: The feature input obtained by the previous SwinTransformer layer is represented by a two-dimensional feature map X, which is divided into non-overlapping windows of size μ×μ. Local self-attention is calculated for each feature in each window, and relative position encoding is applied to the attention module, which is specifically expressed as: Q=X i P Q K=X i P K V=X i P V in, is the projection matrix shared across different windows, i is the index of the feature of the i-th local window, B represents the relative position deviation containing the learnable parameters, and C represents the number of channels; For a given local window feature from the n-1th Swin Transformer layer The calculation of the current SwinTransformer layer is expressed as: Among them, LN represents layer normalization, MSA and MLP represent multi-head self-attention and multi-layer perceptron respectively. and Respectively represent the output of the multi-layer perceptron and multi-head self-attention after residual connection; Add a normalization layer before the multi-layer perceptron and multi-head self-attention.

5. The method for removing multiple severe weather conditions from an image according to claim 1, wherein: The first inherent feature fusion block is based on a convolutional layer with LeakyReLU and uses residual connections to complete the channel interaction and fusion of inherent shallow features and inherent deep features to generate a new feature representation f out , specifically expressed as: f out =IFF(f S +f D )=LR(Conv(f S +f D )) Among them, IFF(·) represents the first intrinsic feature fusion block, LR(·) represents the LeakyReLU activation function, Conv represents convolution, and f S represents the inherent shallow features, f D Indicates inherent deep features.

6. The method for removing multiple severe weather conditions from an image according to claim 1, wherein: The first high-resolution image reconstruction module obtains the enhanced feature representation generated by the first intrinsic feature fusion block, learns a set of upsampling filters based on the sub-pixel convolution layer, and reconstructs the target image under the supervision of the high-resolution true value, which is specifically expressed as follows: Where PS(·) represents the pixel shuffling operation, HRIR(·) represents the first high-resolution image reconstruction module, and f out It represents the enhanced feature representation generated by the first intrinsic feature fusion block, and Conv represents convolution.

7. The method for removing multiple severe weather conditions from an image according to claim 1, wherein: Based on multiple degradation robust losses, the backbone network and auxiliary branches are trained collaboratively to obtain a trained multiple severe weather removal model, which specifically includes: Obtaining a degraded image sample to be repaired and a clean image sample corresponding to the degraded image sample to be repaired; After downsampling the degraded image samples to be repaired, they are input into the backbone network to extract inherent shallow features and inherent deep features. The inherent shallow and deep features are interacted and fused to generate enhanced feature representations, and the repaired image corresponding to the degraded image to be repaired is reconstructed; The inherent shallow and deep features are interacted and fused to reconstruct the degraded output and output a high-resolution degraded image; Introducing multiple degradation robust losses to constrain the feature extraction process through inherent shallow feature extraction blocks with shared weights; The predicted restored image is compared with the corresponding clean image sample until the multiple severe weather removal model is determined to have converged, and the iterative training process ends.

8. The method for removing multiple severe weather conditions from an image according to claim 1, wherein: In the collaborative training process of the backbone network and the auxiliary branches, select N d For degraded images and the repaired image Form a batch of positive samples, denoted as The positive samples are input into the batch inherent shallow feature extraction block to obtain the corresponding shallow feature set; Using an additional 2(N d -1) samples form a negative sample batch and input the inherent shallow feature extraction block to obtain the corresponding shallow feature set.

9. The method for removing multiple severe weather conditions from an image according to claim 1, wherein: In the collaborative training process of the backbone network and auxiliary branches, the overall loss function includes: pixel loss function, perceptual loss function, negative structural similarity loss function and multiple degradation robust loss function, which can be specifically expressed as: in, is the pixel loss function, I′ l is the image sample to be repaired, is the reconstructed degraded image sample corresponding to the image sample to be repaired, I l is the clean image sample corresponding to the image sample to be repaired, is the restored image sample corresponding to the image sample to be restored, represents element-wise subtraction, represents the mean operation, is the perceptual loss function, φ q represents the operation of extracting features from the input image, is the negative structural similarity loss function, SSIM represents the structural similarity calculation function, is the multiple degradation robust loss function, Refers to the feature set extracted from other degraded images and their corresponding real images in the same batch, N d Indicates the number of sample pairs, sim cos Represents the cosine similarity between two features.

10. A system for removing multiple bad weather images, characterized in that: Used to implement the method for removing multiple severe weather conditions from an image as described in any one of claims 1 to 9, the system comprises: an image acquisition module to be repaired, a downsampling module, a model building module, a collaborative training module, and a repaired image output module; The image acquisition module to be restored is used to acquire images to be restored, including degraded images affected by multiple concurrent severe weather conditions; The downsampling module is used to downsample the image to be repaired; The model building module is used to build a multiple severe weather removal model, input the downsampled image to be repaired into the multiple severe weather removal model, and the multiple severe weather removal model removes multiple concurrent severe weather events in the degraded image to be repaired to generate a repaired degraded image; The multiple severe weather removal model includes a backbone network, auxiliary branches and multiple degradation robust losses; The backbone network includes an inherent shallow feature extraction block, an inherent deep feature extraction block, a first inherent feature fusion block, and a first high-resolution image reconstruction module. The inherent shallow feature extraction block is used to extract inherent shallow features of the downsampled image to be repaired, the inherent deep feature extraction block is used to extract inherent deep features of the inherent shallow features, the first inherent feature fusion block is used to interact and fuse the inherent shallow features and the inherent deep features across channels to generate an enhanced feature representation, and the first high-resolution image reconstruction module is used to extract effective feature information under the supervision of high-resolution true values ​​and reconstruct a repaired image corresponding to the degraded image to be repaired; The auxiliary branch includes a second intrinsic feature fusion block and a second high-resolution image reconstruction module, wherein the second intrinsic feature fusion block is used to interact and fuse the intrinsic shallow features and the intrinsic deep features across channels, and the second high-resolution image reconstruction module is used to reconstruct the degraded output under the supervision of the high-resolution degraded input image and output the high-resolution degraded image; The collaborative training module is used to collaboratively train the backbone network and the auxiliary branches based on multiple degradation robust losses to obtain a trained multiple severe weather removal model; The repaired image output module is used to input the degraded image to be repaired into the trained multiple severe weather removal model to obtain a repaired image.

Images