Method and system for removing image in multiple severe weathers

Through the coordinated training of the downsampling-removal-reconstruction process and the multiple inclement weather removal model, the problem of multiple concurrent inclement weather removal in the existing technology is solved, the image repair quality is improved, and efficient removal of multiple concurrent inclement weather is achieved.

CN118967520AActive Publication Date: 2024-11-15GUANGZHOU UNIVERSITY

Patent Information

Application Number
CN202410954505.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-17
Publication Date
2024-11-15
Estimated Expiration
2044-07-17

AI Technical Summary

Technical Problem

The prior art is difficult to effectively remove the degradation of images by multiple concurrent bad weather, and existing methods usually require specific model training for different types of bad weather, which is inefficient and difficult to adapt to complex concurrent scenarios.

Method used

The downsampling-removal-reconstruction process is adopted to construct multiple bad weather removal models, use the backbone network and auxiliary branches for collaborative training, and combine multiple degradation robust losses to compress the constituent particles of bad weather, reduce absolute perturbation, and improve image repair quality.

Benefits of technology

It greatly reduces the difficulty of removing multiple concurrent bad weather, improves the quality of image repair, enhances the removal processing performance of multiple concurrent bad weather, and can effectively handle the removal task of multiple concurrent bad weather in the image.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118967520A_ABST
    Figure CN118967520A_ABST
Patent Text Reader

Abstract

The invention discloses a method and a system for removing images in multiple severe weathers, and the method comprises the following steps: obtaining to-be-restored images, including degraded images affected by multiple concurrent severe weathers; carrying out down-sampling on the to-be-restored image; constructing a multi-severe weather removal model, inputting the down-sampled to-be-restored image into the multi-severe weather removal model, and removing multiple concurrent severe weather in the to-be-restored degraded image by the multi-severe weather removal model to generate a restored degraded image; based on multiple degradation robust loss, performing cooperative training on the backbone network and the auxiliary branches to obtain a trained multiple severe weather removal model; and the degraded image to be restored is based on the trained multiple severe weather removal model to obtain a restored image. According to the method, a down-sampling-removal-reconstruction process multiple concurrent severe weather removal normal form is constructed, the difficulty of removing multiple concurrent severe weather is greatly reduced, and the image restoration quality is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of image restoration, and in particular to a method and system for removing multiple severe weather conditions from an image. Background Art

[0002] Image inpainting is a low-level vision task, including deblurring, denoising, super-resolution, etc., which aims to restore or reconstruct high-quality versions from low-quality images, thereby improving human visual perception and the performance of many outdoor vision systems. The imaging quality of scenes in outdoor environments is inevitably affected by various severe weather conditions (such as rain, fog, snow, and raindrops). Therefore, removing the degradation caused by severe weather has gradually become a special image inpainting task and has attracted widespread attention in computer vision research. Recently, some studies have shown significant results in severe weather removal, but they all target a single type of severe weather and are difficult to transfer to other severe weather. To solve this problem, some methods have attempted to develop a framework to remove multiple degradations, but they usually require specific model training for different degradations, that is, training a set of model parameters for each degradation, which is inefficient and limited in practical applications.

[0003] In bad weather removal and other restoration tasks such as deblurring and denoising, each image degradation (e.g., bad weather, blur, noise, etc.) presents its own unique perturbation pattern that needs to be processed or restored. Moreover, in the real world, it is too ideal to remove bad weather in outdoor images that are only affected by a single type of degradation, as multiple bad weather conditions often co-occur in the same scene. However, due to the significant differences in feature distributions between different degradation types, the above methods often perform poorly in this case, because modeling the relationship between various degradation forms becomes particularly challenging when dealing with concurrent complex degradation scenarios. Summary of the invention

[0004] In order to overcome the defects and shortcomings of the prior art, the present invention provides a method and system for removing multiple severe weather events in an image. The present invention constructs a multiple concurrent severe weather removal paradigm in a downsampling-removal-reconstruction process, and applies downsampling technology to the removal of multiple concurrent severe weather events, thereby compressing the constituent particles of different severe weather events and reducing the absolute disturbance caused by degradation.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] The present invention provides a method for removing multiple severe weather conditions from an image, comprising the following steps:

[0007] Acquire images to be restored, including degraded images affected by multiple concurrent severe weather events;

[0008] Downsample the image to be repaired;

[0009] Constructing a multiple bad weather removal model, inputting the downsampled image to be repaired into the multiple bad weather removal model, wherein the multiple bad weather removal model removes multiple concurrent bad weather in the degraded image to be repaired, and generates a repaired degraded image;

[0010] The multiple severe weather removal model includes a backbone network, auxiliary branches and multiple degradation robust losses;

[0011] The backbone network includes an inherent shallow feature extraction block, an inherent deep feature extraction block, a first inherent feature fusion block and a first high-resolution image reconstruction module, wherein the inherent shallow feature extraction block is used to extract the inherent shallow features of the downsampled image to be repaired, the inherent deep feature extraction block is used to extract the inherent deep features of the inherent shallow features, the first inherent feature fusion block is used to interact and fuse the inherent shallow features and the inherent deep features across channels to generate an enhanced feature representation, and the first high-resolution image reconstruction module is used to extract effective feature information under the supervision of high-resolution true values, and reconstruct a repaired image corresponding to the degraded image to be repaired;

[0012] The auxiliary branch includes a second intrinsic feature fusion block and a second high-resolution image reconstruction module, wherein the second intrinsic feature fusion block is used to interact and fuse the intrinsic shallow features and the intrinsic deep features across channels, and the second high-resolution image reconstruction module is used to reconstruct the degraded output under the supervision of the high-resolution degraded input image and output the high-resolution degraded image;

[0013] Based on multiple degradation robust losses, the backbone network and auxiliary branches are trained collaboratively to obtain a trained multiple severe weather removal model;

[0014] The degraded image to be restored is restored based on the trained multiple severe weather removal model.

[0015] As a preferred technical solution, the inherent shallow feature extraction block uses a convolutional layer to extract the inherent shallow feature f S , specifically expressed as:

[0016] f S =IFE S (I′ l ↓)=Conv(I′ l ↓)

[0017] Among them, I′ l ↓ indicates the image to be repaired after downsampling, IFE S(.) represents the inherent shallow feature extraction block, and Conv(.) represents the convolutional layer, which maps the downsampled image space to be repaired to a higher-dimensional feature space.

[0018] As a preferred technical solution, the inherent deep feature extraction block includes several residual Swin Transformers and a convolutional layer, each residual Swin Transformer includes several Swin Transformer layers and a convolutional layer, and the output of the nth Swin Transformer layer is expressed as:

[0019] f m,n =STL m,n (f m,n-1 )

[0020] Among them, f m,n Represents the intermediate features obtained from the nth Swin Transformer layer, STL m,n (·) represents the nth Swin Transformer layer in the mth residual Swin Transformer, f m,n-1 represents the intermediate features obtained from the n-1th SwinTransformer layer;

[0021] A convolutional layer is added before the residual connection, and the residual Swin Transformer is expressed as:

[0022] f m =RSTB m (f m-1 )=Conv(f m,N )+f m,0

[0023] Among them, f m represents the output of the mth residual Swin Transformer, f m-1 represents the output of the m-1th residual SwinTransformer, f m,N represents the output of the Nth Swin Transformer layer in the mth residual Swin Transformer, f m,0 Represents the input of the mth residual Swin Transformer. The inherent shallow features are input into the residual Swin Transformer to extract the inherent deep features. RSTB m represents the mth residual Swin Transformer, Conv represents convolution;

[0024] The inherent deep features are obtained through several Swin Transformer layers and finally a convolutional layer.

[0025] As a preferred technical solution, the feature input obtained by the previous Swin Transformer layer is represented by a two-dimensional feature map X, the two-dimensional feature map X is divided into non-overlapping windows of size μ×μ, local self-attention calculation is performed on each feature in each window, and relative position encoding is applied to the attention module, which is specifically expressed as:

[0026]

[0027] Q=X i P Q

[0028] K=X i P K

[0029] V=X i P V

[0030] Among them, P Q , P K , is the projection matrix shared across different windows, i is the index of the feature of the i-th local window, B represents the relative position deviation containing the learnable parameters, and C represents the number of channels;

[0031] For a given local window feature from the n-1th Swin Transformer layer The calculation of the current SwinTransformer layer is expressed as:

[0032]

[0033] Among them, LN represents layer normalization, MSA and MLP represent multi-head self-attention and multi-layer perceptron respectively. and They represent the outputs of the multi-layer perceptron and multi-head self-attention after residual connection respectively;

[0034] Add a normalization layer before the multi-layer perceptron and multi-head self-attention.

[0035] As a preferred technical solution, the first inherent feature fusion block is based on a convolutional layer with LeakyReLU, and uses residual connection to complete the channel interaction and fusion of inherent shallow features and inherent deep features to generate a new feature representation f out , specifically expressed as:

[0036] f out =IFF(fS +f D )=LR(Conv(f S +f p ))

[0037] Among them, IFF(·) represents the first intrinsic feature fusion block, LR(·) represents the LeakyReLU activation function, Conv represents convolution, and f S represents the inherent shallow features, f D Indicates inherent deep features.

[0038] As a preferred technical solution, the first high-resolution image reconstruction module obtains the enhanced feature representation generated by the first intrinsic feature fusion block, learns a set of upsampling filters based on the sub-pixel convolution layer, and reconstructs the target image under the supervision of the high-resolution true value, which is specifically expressed as follows:

[0039]

[0040] Where PS(·( represents the pixel shuffle operation, HRIR(·) represents the first high-resolution image reconstruction module, and f out It represents the enhanced feature representation generated by the first intrinsic feature fusion block, and Conv represents convolution.

[0041] As a preferred technical solution, based on multiple degradation robust losses, the backbone network and the auxiliary branches are collaboratively trained to obtain a trained multiple severe weather removal model, which specifically includes:

[0042] Obtaining a degraded image sample to be repaired and a clean image sample corresponding to the degraded image sample to be repaired;

[0043] After downsampling the degraded image samples to be repaired, they are input into the backbone network to extract inherent shallow features and inherent deep features, interact and fuse the inherent shallow and deep features to generate enhanced feature representation, and reconstruct the repaired image corresponding to the degraded image to be repaired;

[0044] The inherent shallow and deep features are interacted and fused to reconstruct the degraded output and output a high-resolution degraded image;

[0045] Introducing multiple degradation robust losses to constrain the feature extraction process through inherent shallow feature extraction blocks with shared weights;

[0046] The predicted restored image is compared with the corresponding clean image sample until the multiple severe weather removal model is determined to converge, and the iterative training process ends.

[0047] As a preferred technical solution, in the collaborative training process of the backbone network and the auxiliary branches, select N d For degraded images and the repaired image Form a batch of positive samples, denoted as

[0048] The positive sample pairs are input into the inherent shallow feature extraction block of the batch to obtain the corresponding shallow feature set;

[0049] Using an additional 2(N d -1( samples form a negative sample batch and input into the inherent shallow feature extraction block to obtain the corresponding shallow feature set.

[0050] As a preferred technical solution, in the collaborative training process of the backbone network and the auxiliary branches, the overall loss function includes: pixel loss function, perceptual loss function, negative structural similarity loss function and multiple degradation robust loss function, which are specifically expressed as:

[0051]

[0052] in, is the pixel loss function, I′ l is the image sample to be repaired, is the reconstructed degraded image sample corresponding to the image sample to be repaired, I l is the clean image sample corresponding to the image sample to be repaired, is the restored image sample corresponding to the image sample to be restored, represents element-wise subtraction, represents the mean operation, is the perceptual loss function, φ q represents the operation of extracting features from the input image, is the negative structural similarity loss function, SSIM represents the structural similarity calculation function, is the multiple degradation robust loss function, Refers to the feature set extracted from other degraded images and their corresponding real images in the same batch, N d Indicates the number of sample pairs, sim cos Represents the cosine similarity between two features.

[0053] The present invention also provides a system for removing multiple severe weather conditions from images, which is used to implement the above-mentioned method for removing multiple severe weather conditions from images. The system comprises: an image acquisition module to be repaired, a downsampling module, a model building module, a collaborative training module, and a repaired image output module;

[0054] The to-be-restored image acquisition module is used to acquire the to-be-restored image, including a degraded image affected by multiple concurrent severe weather conditions;

[0055] The downsampling module is used to downsample the image to be repaired;

[0056] The model building module is used to build a multiple bad weather removal model, input the downsampled image to be repaired into the multiple bad weather removal model, and the multiple bad weather removal model removes multiple concurrent bad weather in the degraded image to be repaired to generate a repaired degraded image;

[0057] The multiple severe weather removal model includes a backbone network, auxiliary branches and multiple degradation robust losses;

[0058] The backbone network includes an inherent shallow feature extraction block, an inherent deep feature extraction block, a first inherent feature fusion block and a first high-resolution image reconstruction module, wherein the inherent shallow feature extraction block is used to extract the inherent shallow features of the downsampled image to be repaired, the inherent deep feature extraction block is used to extract the inherent deep features of the inherent shallow features, the first inherent feature fusion block is used to interact and fuse the inherent shallow features and the inherent deep features across channels to generate an enhanced feature representation, and the first high-resolution image reconstruction module is used to extract effective feature information under the supervision of high-resolution true values, and reconstruct a repaired image corresponding to the degraded image to be repaired;

[0059] The auxiliary branch includes a second intrinsic feature fusion block and a second high-resolution image reconstruction module, wherein the second intrinsic feature fusion block is used to interact and fuse the intrinsic shallow features and the intrinsic deep features across channels, and the second high-resolution image reconstruction module is used to reconstruct the degraded output under the supervision of the high-resolution degraded input image and output the high-resolution degraded image;

[0060] The collaborative training module is used to collaboratively train the backbone network and the auxiliary branches based on multiple degradation robust losses to obtain a trained multiple severe weather removal model;

[0061] The repaired image output module is used to input the degraded image to be repaired into the trained multiple severe weather removal model to obtain a repaired image.

[0062] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0063] (1) The present invention constructs a downsampling-removal-reconstruction process, which compresses the constituent particles of severe weather and reduces the absolute disturbance caused by various degradation phenomena, thereby greatly reducing the difficulty of removing multiple concurrent severe weather and improving the quality of image restoration. The multiple severe weather removal model can bring a powerful performance improvement to the removal of multiple concurrent severe weather in an image.

[0064] (2) The present invention downsamples the degraded image affected by severe weather through a bicubic interpolation algorithm to obtain a low-resolution version thereof and inputs the low-resolution version into the backbone network. The low-resolution version is processed by an inherent shallow feature extraction block and an inherent deep feature extraction block to extract inherent features. In order to better guide feature learning, a first inherent feature fusion block is used to interact and fuse the inherent shallow and deep features according to their channels to generate a new feature representation. The first high-resolution image reconstruction module uses the above feature output to generate a repaired degraded image corresponding to the degraded image to be repaired. At the same time, an auxiliary branch equipped with a second inherent feature fusion block and a second high-resolution image reconstruction module optimizes the inherent features through fine-grained structural representation under the supervision of high-resolution degraded input, thereby guiding and optimizing the learning of feature representation.

[0065] (3) In order to ensure robustness to multiple concurrent degradation phenomena, a multiple degradation robust loss is introduced to better constrain the feature extraction process through inherent shallow feature extraction blocks with shared weights to form collaborative training. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 It is a flow chart of the method for removing multiple severe weather conditions from an image according to the present invention;

[0067] Figure 2 A schematic diagram of the network architecture of the multiple severe weather removal model of the present invention;

[0068] Figure 3 (a) is a schematic diagram of the network architecture of the residual Swin Transformer of the present invention;

[0069] Figure 3 (b) is a schematic diagram of the network architecture of the Swin Transformer layer of the present invention. DETAILED DESCRIPTION

[0070] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0071] Example 1

[0072] like Figure 1 As shown, this embodiment provides a method for removing multiple severe weather conditions from an image, comprising the following steps:

[0073] S1: Acquire an image to be restored, wherein the image to be restored includes a degraded image affected by multiple concurrent severe weather conditions;

[0074] S2: down-sample the image to be repaired;

[0075] Specifically, the image to be repaired is downsampled using a bicubic interpolation algorithm to obtain its low-resolution version;

[0076] S3: construct a multiple bad weather removal model, input the image to be repaired into the multiple bad weather removal model, so that the multiple bad weather removal model removes multiple concurrent bad weather in the degraded image to be repaired, and generates a clean repaired degraded image;

[0077] like Figure 2 As shown, the multiple severe weather removal model includes a backbone network, auxiliary branches and multiple degradation robust losses. The backbone network is used to extract the inherent features of the degraded image, interact and fuse the inherent shallow and deep features, and generate a repaired clean image corresponding to the degraded image to be repaired through the generated new feature representation.

[0078] Specifically, the backbone network includes an inherent shallow feature extraction block, an inherent deep feature extraction block, a first inherent feature fusion block and a first high-resolution image reconstruction module. The inherent shallow feature extraction block and the inherent deep feature extraction block are used to process the downsampled degraded input and convert it into features with semantic meaning. The first inherent feature fusion block is used for inter-channel interaction and fusion of features to generate enhanced feature representation. The first high-resolution image reconstruction module, under the supervision of high-resolution true values, uses the extracted effective feature information to reconstruct a repaired clean image corresponding to the degraded image to be repaired.

[0079] In this embodiment, the intrinsic shallow feature extraction block applies a simple convolutional layer to extract the intrinsic shallow features f S , the formula is as follows:

[0080] f S =IFE S (I l ′ ↓)=Conv(I l ′ ↓)

[0081] Among them, I′ l ↓ represents the degraded input after downsampling, IFE S (.) is an inherent shallow feature extraction block, specifically a convolutional layer, which can map the input image space to a higher-dimensional feature space, thereby improving the optimization stability without increasing unnecessary computational costs, and is more consistent with the design of the model in this embodiment.

[0082] Since the residual Swin Transformer block performs well in capturing long-distance dependencies. In the scheme of this embodiment, thanks to downsampling, the residual Swin Transformer method based on local windows is applied on low-resolution feature maps, which can be processed on a larger receptive field, allowing the network to better learn long-distance dependencies. Based on these considerations, the residual Swin Transformer will be used as a block suitable for the scheme of this embodiment to perform local attention and cross-window interaction, thereby constructing an inherent deep feature extraction block. The inherent shallow features are further extracted by the inherent deep feature extraction block to obtain the inherent deep features f D , the formula is as follows:

[0083] f D =IFE D (f S )

[0084] Among them, IFE D (·) is the intrinsic deep feature extraction block, which consists of several residual Swin Transformers and a convolutional layer; Figure 3 (a) Figure 3 As shown in (b), the residual Swin Transformer is a residual block that integrates several Swin Transformer layers and a convolutional layer. In the mth residual Swin Transformer, given the input feature f m,0 , the output of the nth Swin Transformer layer is expressed as:

[0085] f m,n =STL m,n (f m,n-1 ),n∈{1,2,…,N}

[0086] Among them, f m,n Represents the intermediate features obtained from the nth Swin Transformer layer, STL m,n (·) is the nth Swin Transformer layer in the mth residual Swin Transformer, f m,n-1 represents the intermediate features obtained from the n-1th SwinTransformer layer, and N represents the number of Swin Transformer layers. In order to enhance translation equivariance and achieve aggregation of features at different levels, a convolutional layer is added before the residual connection. Mathematically, the residual SwinTransformer can be defined as:

[0087] f m =RSTB m (f m-1)=Conv(f m,N )+f m,0 ,m∈{1,2,…,M}

[0088] Among them, f m is the output of the mth residual Swin Transformer, f m-1 represents the output of the m-1th residual SwinTransformer, f m,N represents the output of the Nth Swin Transformer layer in the mth residual Swin Transformer, f m,0 Represents the input of the mth residual Swin Transformer, the inherent shallow feature f S The input residual SwinTransformer is used for further intrinsic deep feature extraction. The intrinsic deep features obtained in this way contain high-frequency information, which is crucial for pixel recovery and detail preservation. M represents the number of residual Swin Transformers, RSTB m Represents the mth residual Swin Transformer.

[0089] In particular, the Swin Transformer layer performs self-attention within non-overlapping local windows. Given a 2D feature map X, i.e., the feature input obtained from the previous Swin Transformer layer, represented by a 2D feature map X, X is divided into non-overlapping windows of size μ×μ, and then features are extracted from each window i Where C represents the number of channels. Next, the Swin Transformer layer performs local self-attention calculations on each feature within each window and applies relative position encoding to the attention module. i The calculation formula of the attention mechanism is:

[0090] Q=X i P Q , K = X i P K , V = X i P V ,

[0091]

[0092] Among them, P Q , P K , is the projection matrix shared across different windows, where C represents the number of channels. i∈{1,2,…,I} is the index of the i-th local window feature. B represents the relative position bias containing learnable parameters. The Swin Transformer layer in the residual Swin Transformer includes two core components: multi-layer perceptron and multi-head self-attention, as shown in Figure 3 (a) Figure 3 (b) shows that for a given local window feature from the n-1th Swin Transformer layer The calculation of the current Swin Transformer layer is expressed as:

[0093]

[0094]

[0095] Among them, LN represents layer normalization, MSA and MLP represent multi-head self-attention and multi-layer perceptron respectively. and They represent the outputs of the multi-layer perceptron and multi-head self-attention after residual connection. A layer normalization layer is added before the multi-layer perceptron and multi-head self-attention. The Swin Transformer layer performs self-attention design in a local window, which significantly reduces the computational complexity compared to global self-attention; the inherent deep features f are obtained through several Swin Transformer layers and finally through a convolutional layer. D .

[0096] In this embodiment, the first intrinsic feature fusion block more effectively mines the information of image reconstruction and enhances the modeling ability of potential feature distribution. It uses a convolutional layer with LeakyReLU and adopts residual connections to promote cross-channel interaction and fusion of shallow and deep features to generate a new feature representation f out , the formula is as follows:

[0097] f out =IFF(f S +f D )=LR(Conv(f S +f D ))

[0098] Among them, IFF(·) represents the first intrinsic feature fusion block, LR(·) is the LeakyReLU activation function, and through a long skip connection, the shallow feature f S Directly input into the first intrinsic feature fusion block to preserve low-frequency information.

[0099] In this embodiment, the first high-resolution image reconstruction module converts the final feature f obtained from the first intrinsic feature fusion block into out , using a sub-pixel convolutional layer, which learns a set of upsampling filters to upsample the final low-resolution feature map to a high-resolution clean output Under the supervision of the high-resolution true value, the first high-resolution image reconstruction module reconstructs the clean target and promotes the recovery of more details. The process can be summarized as follows:

[0100]

[0101] Among them, PS(·) represents the pixel shuffling operation, and HRIR(·) represents the first high-resolution image reconstruction module. In the first high-resolution image reconstruction module, two convolutional layers are designed at the beginning and end respectively. The former is intended to increase the number of channels and provide enough data for the subsequent pixel shuffling to effectively achieve upsampling. The latter adjusts and refines the feature map after upsampling to ensure that the output image has the correct number of channels.

[0102] Combination Figure 2 As shown, the auxiliary branch includes a second intrinsic feature fusion block and a second high-resolution image reconstruction module, which are used to optimize the intrinsic features through fine-grained structure representation, aiming to guide and optimize the learning of feature representation;

[0103] Specifically, by adopting downsampling techniques to process images under a variety of adverse weather conditions, the compositional particles present in these conditions are effectively mitigated from a low-resolution perspective and the absolute perturbations caused by degradation are reduced. While the backbone network is able to reconstruct high-resolution clean outputs from low-resolution degraded inputs, an auxiliary branch is designed to enhance the ability to capture local contextual information, which makes the reconstruction of high-resolution details more refined, overcoming the challenges posed by limited effective image information. The auxiliary branch includes a second intrinsic feature fusion block and a second high-resolution image reconstruction module, which do not share weights with the corresponding modules in the backbone network.

[0104] The input of the auxiliary branch is the result of concat- ing the inherent shallow features and the inherent deep features. The second inherent feature fusion block utilizes a convolutional layer with LeakyReLU and adopts residual connection to promote cross-channel interaction and fusion of inherent shallow features and inherent deep features. The second high-resolution image reconstruction module reconstructs the degraded output under the supervision of the high-resolution degraded input image, and the output is a reconstructed high-resolution degraded image, thereby effectively realizing the reconstruction of fine-grained structural information of the image under low-resolution input.

[0105] The limitation of capturing internal pixel relationships is addressed by guiding and optimizing feature representation learning through auxiliary branches. As a result, the model is able to effectively eliminate multiple degradations while preserving reconstruction quality, and demonstrates attention to detail and accurate modeling even with only low-resolution severe weather images. To avoid additional computational cost, the auxiliary branches are removed during the inference phase.

[0106] S4: Based on multiple degradation robust losses, the backbone network and auxiliary branches are co-trained to obtain the trained multiple severe weather removal model;

[0107] In this embodiment, collaborative training is performed by introducing contrast constraints between the backbone network and the auxiliary branches, and multiple degradation robust losses are used to better constrain the feature extraction process through inherent shallow feature extraction blocks with shared weights, forming collaborative training.

[0108] In this embodiment, the backbone network is trained under clean labeled data, while the auxiliary branches are trained under supervision with degraded original (non-downsampled) input. For concurrent multiple severe weather images, it is necessary to ensure that the areas not covered by severe weather can be well restored during the training process. The auxiliary branches are added to ensure that the details of these uncovered areas are consistent in restoration, and the image has limited effective pixels, so that the limited pixel information can be utilized to the greatest extent.

[0109] In this embodiment, in view of the significant differences in the feature distributions of multiple severe weather elements in the same scene, a multiple degradation robust loss is introduced to promote the learned representation to be robust to multiple concurrent degradation phenomena. By promoting the collaborative training of the backbone network and the auxiliary branches, the outputs of the backbone network and the auxiliary branches share common properties and the embedding in the latent space should also have a lower distance. Therefore, contrast constraints are imposed in the shallow feature space, and appropriate positive and negative pairs are constructed, further taking advantage of the learned related feature distribution to obtain sufficient information to reconstruct a clean background.

[0110] Specifically, the outputs of the auxiliary branch and the backbone network are mapped to the embedding space through the inherent shallow feature extraction block with shared weights, and the similarity metric is optimized to make them closer. It is expected that the refined features can resist multiple degradations in bad weather images. Specifically, during the training process, N is first selected. d For degraded output and clean output Form a batch of positive samples, denoted as These images are fed into the intrinsic shallow feature extraction block to obtain the shallow feature set To constrain the common features in the embedding, an additional 2(N d-1) samples form a negative sample batch and input them into the intrinsic shallow feature extraction block. By encouraging positive samples to be closer and keep a large distance from negative samples, the constraints of multiple degradation robust losses are imposed, thereby establishing associations between representations with similar feature distributions and contrasting with dissimilar representations. The ultimate goal is to promote the collaboration of different modules and help the intrinsic shallow feature extraction block accurately extract information relevant to image reconstruction.

[0111] In this embodiment, the training process of the multiple severe weather removal models includes:

[0112] Repeat the following training operations of the multiple severe weather removal model until the multiple severe weather removal model is determined to have converged:

[0113] Obtaining a sample of a degraded image to be repaired and a clean image sample corresponding to the sample degraded image to be repaired;

[0114] Before the degraded image samples to be repaired are input into the backbone network, they are first downsampled to obtain their low-resolution versions. The low-resolution degraded images are input into the backbone network and processed by the inherent shallow feature extraction block and the inherent deep feature extraction block to extract the inherent features. Then, they are passed through the first inherent feature fusion block to interact and fuse the inherent shallow and deep features according to their channels to generate new feature representations. Subsequently, the first high-resolution image reconstruction module uses the above feature outputs to reconstruct and generate the repaired clean image corresponding to the degraded image to be repaired. The second inherent feature fusion block interacts and fuses the inherent shallow features and the inherent deep features across channels. The second high-resolution image reconstruction module reconstructs the degraded output under the supervision of the high-resolution degraded input image. The output is a reconstructed high-resolution degraded image, thereby effectively realizing the reconstruction of fine-grained structural information of the image under low-resolution input. In addition, multiple degradation robust losses are introduced to constrain the feature extraction process through the inherent shallow feature extraction blocks with shared weights.

[0115] The predicted restored clean image is compared with the corresponding clean image sample to compare the image prediction accuracy and the image restoration effect. When it is determined that the multiple severe weather removal model has not converged according to the comparison result, the updated sample clean image to be restored and the sample clean image corresponding to the sample degraded image to be restored are obtained.

[0116] In this embodiment, when training the multiple severe weather removal models, it also includes:

[0117] The multiple bad weather removal models are updated according to the overall loss function of the multiple bad weather removal models; wherein the overall loss function is composed of a pixel loss function, a perceptual loss function, a negative structural similarity loss function and a multiple degradation robust loss function;

[0118] The pixel loss function is calculated according to the following formula:

[0119]

[0120] in, is the pixel loss function, I′ l is the image sample to be repaired, is the reconstructed degraded image sample corresponding to the image sample to be repaired, I l is the clean image sample corresponding to the image sample to be repaired, is the sample repaired image corresponding to the sample image to be repaired, represents element-wise subtraction, represents the mean operation;

[0121] The perceptual loss function is calculated according to the following formula:

[0122]

[0123] in, is the perceptual loss function, φ q It represents the operation of extracting features from the input image, specifically the output feature map of the qth layer of a pre-trained convolutional neural network (such as VGG, ResNet, etc.), which is used to map the image to a high-dimensional feature space, and calculate the perceptual loss by comparing the differences between these feature maps. In this embodiment, the conv2,3 layers of the VGG-16 network pre-trained on ImageNet are preferably used to extract features, and the distance between these features is calculated as a measure of perceptual similarity.

[0124] The negative structural similarity loss function is calculated according to the following formula:

[0125]

[0126] in, is the negative structural similarity loss function;

[0127] The multiple degradation robust loss function is calculated according to the following formula:

[0128]

[0129] in, is the multiple degradation robust loss function, Refers to the feature set extracted from other degraded images and their corresponding real images in the same batch, sim cos represents the cosine similarity between two features, and the temperature parameter τ is set to 0.25;

[0130] The overall loss function of the multiple severe weather removal model is calculated according to the following formula:

[0131]

[0132] in, is the overall loss function, λ MDR The first preset parameter, λ p is the second preset parameter.

[0133] S5: The degraded image to be repaired is restored based on the trained multiple bad weather removal model to obtain the restored image.

[0134] The present invention allows for a customized downsampling ratio, thereby achieving a flexible balance between memory requirements and recovery performance. In addition, the model uses upsampling processing (i.e., pixel shuffling operations) in both the first high-resolution image reconstruction module of the backbone network and the second high-resolution image reconstruction module of the auxiliary branch, so that it can not only eliminate the bad weather effects of the low-resolution input, but also reconstruct the high-resolution corresponding image. This feature makes the model highly adaptable to various practical applications that require high-resolution images, which is impossible to achieve with previous methods.

[0135] Example 2

[0136] This embodiment provides a system for removing multiple severe weather conditions from an image, which is used to implement the method for removing multiple severe weather conditions from an image in the above-mentioned embodiment 1. The system includes: an image acquisition module to be repaired, a downsampling module, a model building module, a collaborative training module, and a repaired image output module;

[0137] In this embodiment, the module for acquiring images to be restored is used to acquire images to be restored, including degraded images affected by multiple concurrent severe weather conditions;

[0138] In this embodiment, the downsampling module is used to downsample the image to be repaired;

[0139] In this embodiment, the model building module is used to build a multiple bad weather removal model, and the downsampled image to be repaired is input into the multiple bad weather removal model, and the multiple bad weather removal model removes multiple concurrent bad weather in the degraded image to be repaired to generate a repaired degraded image;

[0140] In this embodiment, the multiple severe weather removal model includes a backbone network, auxiliary branches, and multiple degradation robust losses;

[0141] In this embodiment, the backbone network includes an inherent shallow feature extraction block, an inherent deep feature extraction block, a first inherent feature fusion block and a first high-resolution image reconstruction module, wherein the inherent shallow feature extraction block is used to extract the inherent shallow features of the downsampled image to be repaired, the inherent deep feature extraction block is used to extract the inherent deep features of the inherent shallow features, the first inherent feature fusion block is used to interact and fuse the inherent shallow features and the inherent deep features across channels to generate an enhanced feature representation, and the first high-resolution image reconstruction module is used to extract effective feature information under the supervision of high-resolution true values, and reconstruct a repaired image corresponding to the degraded image to be repaired;

[0142] In this embodiment, the auxiliary branch includes a second intrinsic feature fusion block and a second high-resolution image reconstruction module, wherein the second intrinsic feature fusion block is used to interact and fuse the intrinsic shallow features and the intrinsic deep features across channels, and the second high-resolution image reconstruction module is used to reconstruct the degraded output under the supervision of the high-resolution degraded input image, and output the high-resolution degraded image;

[0143] In this embodiment, the collaborative training module is used to collaboratively train the backbone network and the auxiliary branches based on multiple degradation robust losses to obtain a trained multiple severe weather removal model;

[0144] In this embodiment, the restored image output module is used to input the degraded image to be restored into the trained multiple severe weather removal model to obtain a restored image.

[0145] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.

Claims

1. A method for removing multiple bad weather conditions from an image, characterized in that: The steps include: Acquire images to be restored, including degraded images affected by multiple concurrent severe weather events; Downsample the image to be repaired; Constructing a multiple bad weather removal model, inputting the downsampled image to be repaired into the multiple bad weather removal model, wherein the multiple bad weather removal model removes multiple concurrent bad weather in the degraded image to be repaired, and generates a repaired degraded image; The multiple severe weather removal model includes a backbone network, auxiliary branches and multiple degradation robust losses; The backbone network includes an inherent shallow feature extraction block, an inherent deep feature extraction block, a first inherent feature fusion block and a first high-resolution image reconstruction module, wherein the inherent shallow feature extraction block is used to extract the inherent shallow features of the downsampled image to be repaired, the inherent deep feature extraction block is used to extract the inherent deep features of the inherent shallow features, the first inherent feature fusion block is used to interact and fuse the inherent shallow features and the inherent deep features across channels to generate an enhanced feature representation, and the first high-resolution image reconstruction module is used to extract effective feature information under the supervision of high-resolution true values, and reconstruct a repaired image corresponding to the degraded image to be repaired; The auxiliary branch includes a second intrinsic feature fusion block and a second high-resolution image reconstruction module, wherein the second intrinsic feature fusion block is used to interact and fuse the intrinsic shallow features and the intrinsic deep features across channels, and the second high-resolution image reconstruction module is used to reconstruct the degraded output under the supervision of the high-resolution degraded input image and output the high-resolution degraded image; Based on multiple degradation robust losses, the backbone network and auxiliary branches are trained collaboratively to obtain a trained multiple severe weather removal model; The degraded image to be restored is restored based on the trained multiple severe weather removal model.

2. The method for removing multiple bad weather conditions from an image according to claim 1, characterized in that: The inherent shallow feature extraction block uses a convolutional layer to extract the inherent shallow features f S , specifically expressed as: f S =IFE S (I′ l ↓)=conv(I′ l ↓) Among them, I′ l ↓ indicates the image to be repaired after downsampling, IFE S (.) represents the inherent shallow feature extraction block, and conv(.) represents the convolutional layer, which maps the downsampled image space to be repaired to a higher-dimensional feature space.

3. The method for removing multiple severe weather conditions from an image according to claim 1, characterized in that: The inherent deep feature extraction block includes several residual Swin Transformers and a convolutional layer, each residual Swin Transformer includes several Swin Transformer layers and a convolutional layer, and the output of the nth Swin Transformer layer is expressed as: f m,n =STL m,n (f m,n-1 ) Among them, f m,n Represents the intermediate features obtained from the nth Swin Transformer layer, STL m,n (·) represents the nth Swin Transformer layer in the mth residual Swin Transformer, f m,n-1 represents the intermediate features obtained from the n-1th SwinTransformer layer; A convolutional layer is added before the residual connection, and the residual Swin Transformer is expressed as: f m =RSTB m (f m-1 )=Conv(f m,N )+f m,0 Among them, f m represents the output of the mth residual Swin Transformer, f m-1 represents the output of the m-1th residual SwinTransformer, f m,N represents the output of the Nth Swin Transformer layer in the mth residual Swin Transformer, f m,0 Represents the input of the mth residual Swin Transformer. The inherent shallow features are input into the residual Swin Transformer to extract the inherent deep features. RSTB m represents the mth residual Swin Transformer, Conv represents convolution; The inherent deep features are obtained through several Swin Transformer layers and finally a convolutional layer.

4. The method for removing multiple bad weather conditions from an image according to claim 3, characterized in that: The feature input obtained by the previous SwinTransformer layer is represented by a two-dimensional feature map X, which is divided into non-overlapping windows of size μ×μ. Local self-attention is calculated for each feature in each window, and relative position encoding is applied to the attention module, which is specifically expressed as: Q=X i P Q K=X i P K V=X i P V in, is the projection matrix shared across different windows, i is the index of the feature of the i-th local window, B represents the relative position deviation containing the learnable parameters, and C represents the number of channels; For a given local window feature from the n-1th Swin Transformer layer The calculation of the current SwinTransformer layer is expressed as: Among them, LN represents layer normalization, MSA and MLP represent multi-head self-attention and multi-layer perceptron respectively. and They represent the outputs of the multi-layer perceptron and multi-head self-attention after residual connection respectively; Add a normalization layer before the multi-layer perceptron and multi-head self-attention.

5. The method for removing multiple bad weather conditions from an image according to claim 1, characterized in that: The first inherent feature fusion block is based on a convolutional layer with LeakyReLU and uses residual connections to complete the channel interaction and fusion of inherent shallow features and inherent deep features to generate a new feature representation f out , specifically expressed as: f out =TFF(f S +f D )=LR(Conv(f S +f D )) Among them, IFF(·) represents the first intrinsic feature fusion block, LR(·) represents the LeakyReLU activation function, Conv represents convolution, and f S represents the inherent shallow features, f D Indicates inherent deep features.

6. The method for removing multiple bad weather conditions from an image according to claim 1, characterized in that: The first high-resolution image reconstruction module obtains the enhanced feature representation generated by the first intrinsic feature fusion block, learns a set of upsampling filters based on the sub-pixel convolution layer, and reconstructs the target image under the supervision of the high-resolution true value, which is specifically expressed as: Where PS(·) represents the pixel shuffling operation, HRIR(·) represents the first high-resolution image reconstruction module, and f out It represents the enhanced feature representation generated by the first intrinsic feature fusion block, and Conv represents convolution.

7. The method for removing multiple bad weather conditions from an image according to claim 1, characterized in that: Based on multiple degradation robust losses, the backbone network and auxiliary branches are trained collaboratively to obtain a trained multiple severe weather removal model, which specifically includes: Obtaining a degraded image sample to be repaired and a clean image sample corresponding to the degraded image sample to be repaired; After downsampling the degraded image samples to be repaired, they are input into the backbone network to extract inherent shallow features and inherent deep features, interact and fuse the inherent shallow and deep features to generate enhanced feature representation, and reconstruct the repaired image corresponding to the degraded image to be repaired; The inherent shallow and deep features are interacted and fused to reconstruct the degraded output and output a high-resolution degraded image; Introducing multiple degradation robust losses to constrain the feature extraction process through inherent shallow feature extraction blocks with shared weights; The predicted restored image is compared with the corresponding clean image sample until the multiple severe weather removal model is determined to converge, and the iterative training process ends.

8. The method for removing multiple bad weather conditions from an image according to claim 1, characterized in that: In the collaborative training process of the backbone network and the auxiliary branches, select N d For degraded images and the repaired image Form a batch of positive samples, denoted as The positive sample pairs are input into the inherent shallow feature extraction block of the batch to obtain the corresponding shallow feature set; Using an additional 2(N d -1) samples form a negative sample batch and input them into the inherent shallow feature extraction block to obtain the corresponding shallow feature set.

9. The method for removing multiple bad weather conditions from an image according to claim 1, characterized in that: In the collaborative training process of the backbone network and the auxiliary branches, the overall loss function includes: pixel loss function, perceptual loss function, negative structural similarity loss function and multiple degradation robust loss function, which are specifically expressed as: in, is the pixel loss function, I′ l is the image sample to be repaired, is the reconstructed degraded image sample corresponding to the image sample to be repaired, I l is the clean image sample corresponding to the image sample to be repaired, is the restored image sample corresponding to the image sample to be restored, represents element-wise subtraction, represents the mean operation, is the perceptual loss function, φ q represents the operation of extracting features from the input image, is the negative structural similarity loss function, SSIM represents the structural similarity calculation function, is the multiple degradation robust loss function, Refers to the feature set extracted from other degraded images and their corresponding real images in the same batch, N d Indicates the number of sample pairs, sim cos Represents the cosine similarity between two features.

10. A system for removing multiple bad weather images, characterized in that: Used to implement the method for removing multiple severe weather conditions from an image as described in any one of claims 1 to 9, the system comprises: an image acquisition module to be repaired, a downsampling module, a model building module, a collaborative training module, and a repaired image output module; The to-be-restored image acquisition module is used to acquire the to-be-restored image, including a degraded image affected by multiple concurrent severe weather conditions; The downsampling module is used to downsample the image to be repaired; The model building module is used to build a multiple bad weather removal model, input the downsampled image to be repaired into the multiple bad weather removal model, and the multiple bad weather removal model removes multiple concurrent bad weather in the degraded image to be repaired to generate a repaired degraded image; The multiple severe weather removal model includes a backbone network, auxiliary branches and multiple degradation robust losses; The backbone network includes an inherent shallow feature extraction block, an inherent deep feature extraction block, a first inherent feature fusion block and a first high-resolution image reconstruction module, wherein the inherent shallow feature extraction block is used to extract the inherent shallow features of the downsampled image to be repaired, the inherent deep feature extraction block is used to extract the inherent deep features of the inherent shallow features, the first inherent feature fusion block is used to interact and fuse the inherent shallow features and the inherent deep features across channels to generate an enhanced feature representation, and the first high-resolution image reconstruction module is used to extract effective feature information under the supervision of high-resolution true values, and reconstruct a repaired image corresponding to the degraded image to be repaired; The auxiliary branch includes a second intrinsic feature fusion block and a second high-resolution image reconstruction module, wherein the second intrinsic feature fusion block is used to interact and fuse the intrinsic shallow features and the intrinsic deep features across channels, and the second high-resolution image reconstruction module is used to reconstruct the degraded output under the supervision of the high-resolution degraded input image and output the high-resolution degraded image; The collaborative training module is used to collaboratively train the backbone network and the auxiliary branches based on multiple degradation robust losses to obtain a trained multiple severe weather removal model; The repaired image output module is used to input the degraded image to be repaired into the trained multiple severe weather removal model to obtain a repaired image.

Citation Information

Patent Citations

  • Airborne target approaching guidance system and method based on detection tracking fusion

    CN112364854A

  • Non-ideal supervision-based severe weather image restoration system

    CN117391969A

  • Super-resolution reconstruction method and system for remote sensing image and medium

    CN117788295A

  • Method for detecting traffic signs in severe weather based on composite feature fusion

    CN117789177A

Cited By

  • Image restoration method based on severe weather image restoration network

    CN120031755A

  • An image restoration method based on a severe weather image restoration network

    CN120031755B

  • Image restoration method and system coping with various severe weather environment factors

    CN120355607A

  • An image restoration method and system for coping with various severe weather environment factors

    CN120355607B

  • Target detection method for traffic scene in severe weather based on double-branch fusion

    CN122574791A