A general multi-weather degraded image restoration method

Abstract

A general multi-weather degradation image restoration method comprises the following steps: segmenting a weather degradation image to be processed to obtain a plurality of image blocks, and there is an overlapping area between two adjacent image blocks; performing forward diffusion by adding noise to the image blocks for multiple times to obtain noise blocks; inputting an existing weather degradation image as a condition and a reverse time step into a noise estimation network; generating estimated noise corresponding to the noise blocks by using the noise estimation network, and calculating average estimated noise based on the estimated noise of the overlapping area; performing smoothing processing on the average estimated noise, and performing reverse implicit sampling on the noise blocks based on the average estimated noise after the smoothing processing; and recombining the restoration blocks obtained after the noise blocks are subjected to multiple reverse implicit samplings to obtain a restored weather degradation image. The general multi-weather degradation image restoration method has strong generalization ability and good clarity of the restored image.

Description

Technical Field

[0001] This invention relates to the field of weather-degraded image restoration technology, specifically a general method for restoring multi-weather-degraded images. Background Technology

[0002] Rainy weather can severely interfere with camera imaging, causing adverse effects such as rain fog and rain streaks in images. The presence of rain fog and rain streaks can significantly reduce the performance of subsequent advanced vision tasks. Therefore, image deraining and image dehazing are important tasks in the fields of image processing and computer vision.

[0003] For image deraining tasks, existing methods can be divided into two basic categories: traditional methods and deep learning-based methods. Traditional methods include those based on filters, priors, and models. However, traditional methods are insufficient in handling complex backgrounds and heavily rely on manual design, resulting in insufficient overall efficiency and versatility. Deep learning-based methods significantly outperform traditional methods in performance. The models they generate have excellent image background modeling capabilities, require less human intervention, and greatly improve overall efficiency.

[0004] For the problem of image restoration under various weather conditions, deep learning-based methods can be broadly categorized into three main types: specialized, ensemble, and general. Specialized methods tailor unique network structures for different tasks, resulting in diverse model architectures and parameters, making them difficult to apply to restoration of images with varying degrees of weather degradation. Aggregate methods provide a unified network framework for various tasks; although the model structures are similar, the parameter settings differ, leading to unsatisfactory restoration results. General methods, based on the commonalities of various weather deterioration conditions, construct image restoration networks adaptable to diverse severe weather conditions, and compared to specialized and ensemble methods, can be applied to the restoration of images with various weather degradations.

[0005] In existing technologies, the generative models used in mainstream general-purpose methods mainly include Generative Adversarial Networks (GANs), Variational Autoencoders (VAEs), flow-based models, and Denoising Diffusion Probability Models (DDPMs). Among them, GANs suffer from training instability and mode collapse; VAEs generate samples of lower quality, which may result in blurring and distortion; both flow-based and diffusion models map the data distribution to Gaussian noise, but flow-based models learn an invertible and differentiable function to perform the mapping in a deterministic way, leading to additional constraints on the network structure; DDPMs, due to their learnable forward and backward processes, are relatively more stable in training and have better scalability.

[0006] However, in existing general-purpose multi-weather degraded image restoration methods based on DDPM, the noise estimation network mostly adopts the U-net architecture with CNN as the core. In practical applications, it has been found that there are still many shortcomings, mainly manifested in poor generalization ability. When restoring weather degraded images of different sizes and different degradation forms, the performance is very limited. Summary of the Invention

[0007] To address the shortcomings of existing technologies, this invention provides a general method for restoring degraded images in various weather conditions, which has strong generalization ability and produces images with good clarity.

[0008] To achieve the above objectives, the specific solution adopted by the present invention is as follows:

[0009] A general method for restoring degraded images with multiple weather conditions includes the following steps: The weather degradation image to be processed is segmented into multiple image blocks, and there is an overlapping area between two adjacent image blocks; Noise blocks are obtained by forward diffusion of noise added to image blocks multiple times. Existing weather degradation images are used as conditions and fed into the noise estimation network along with the inverse time step. The noise estimation network consists of multiple basic DiT blocks and optimized DiT blocks that are stacked in an interleaved manner. The optimized DiT blocks are obtained by introducing cascaded context fusion attention based on the basic DiT blocks. A noise estimation network is used to generate estimated noise corresponding to the noise block, and the average estimated noise is calculated based on the estimated noise in the overlapping region. The average estimated noise is smoothed, and the noise block is implicitly sampled in reverse based on the smoothed average estimated noise. The restored weather degradation image is obtained by reconstructing the restored blocks obtained by repeatedly performing reverse implicit sampling on the noise blocks.

[0010] As a further optimization of the aforementioned general multi-weather degraded image restoration method, the specific methods for segmenting the weather degraded image to be processed into multiple image blocks include: Starting from one of the vertexes of the weather degradation image to be processed, determine a grid of size r×r; The grid is moved step by step with a step size of r until the weather degradation image to be processed is decomposed into multiple grids; The weather degradation image to be processed is segmented based on a grid, forming multiple image blocks, with at least one overlapping grid between two adjacent image blocks.

[0011] As a further optimization of the aforementioned general multi-weather degraded image restoration method, the method of adding noise to image blocks multiple times is as follows: Where x0 is the image block, t is the forward time step, and α t Let ò be a predefined noise sequence, where ò is noise sampled from a standard normal distribution.

[0012] As a further optimization of the aforementioned general multi-weather degraded image restoration method, specific methods for generating estimated noise using a noise estimation network include: The existing weather degradation images used as conditions are converted into token vectors of quantity T and dimension D. The size of the token vectors is N×T×D, where N is the batch size, D is the length of the token vector, and T = (H×W) / p. 2 p is a hyperparameter; the reverse time step t' is embedded into all tokens; The token is processed one by one using the basic DiT block and the optimized DiT block to generate estimated noise.

[0013] As a further optimization of the aforementioned general multi-weather degraded image restoration method, the optimized DiT block generation method includes: The attention heads of the multi-head self-attention blocks in the basic DiT block are divided into multiple groups, and the multiple groups of attention heads are distributed sequentially. Introduce a contextual attention mechanism between two adjacent sets of attention heads; Multiple attention heads are concatenated based on the contextual attention mechanism to obtain a concatenated contextual fusion attention block; An optimized DiT block is obtained by replacing the multi-head self-attention block in the basic DiT block with a cascaded context fusion attention block.

[0014] As a further optimization of the aforementioned general multi-weather degraded image restoration method, the computation process of the cascaded context fusion attention block includes: The input feature X in tensor form is split into H feature heads along the feature dimension. Feats_in = split(X,H)∈R H×B×N×(D / H) ,in N is the batch size, T is the sequence length, and D is the feature dimension; Perform CFA and cascade operations on each feature head i. Feat i =CFA(Feat) i ); Reassemble the processed feature headers X_out=concat(Feat0,Feat1,...,Feat H-1)∈R N×T×D ; Merge all recombined feature headers X-out = Fusion(X-out) ∈ R N×T×D .

[0015] As a further optimization of the aforementioned general multi-weather degraded image restoration method, the specific process of CFA operation includes: Receive input data in tensor form Where B is the batch size, N is the sequence length, and C is the feature dimension; Calculate Query(Q), Key matrix K1, and Value(V), with the results in matrix form: Concatenate K1 with the input tensor X along the feature dimension: Merge the spliced ​​matrix Y1 back into the original dimension: Calculate attention weights: Calculate the new Key matrix K2: Calculate the output matrix:

[0016] As a further optimization of the aforementioned general multi-weather degraded image restoration method, the specific method for calculating the average estimated noise based on the estimated noise of overlapping regions includes: Construct a matrix of the same size as the weather degradation image to be processed, and use the estimated noise corresponding to the pixels in the overlapping region as the elements; Calculate the sum of all matrices to obtain Based on the estimated noise level corresponding to each pixel in the overlapping region... Normalization is performed to obtain the average estimated noise.

[0017] Beneficial effects: Based on the denoising diffusion probability model, this invention uses existing weather degradation images as conditions to guide the backsampling process to generate clear background images; by segmenting image blocks, this invention can meet the restoration requirements of weather degradation images of any size, thus improving generalization ability; by introducing cascaded context fusion attention in a cascaded manner to calculate the context fusion attention mechanism of different heads, it can fully capture global and local feature information, further improving the model's generalization ability on various weather degradation images. Attached Figure Description

[0018] Figure 1 This is an overall schematic diagram of the present invention; Figure 2 This is a schematic diagram of the noise estimation network structure; Figure 3 This is a diagram illustrating the token generation process; Figure 4 This is a schematic diagram illustrating the optimized generation method of DiT blocks; Figure 5 This is a schematic diagram of the CFA calculation process. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] Please see Figure 1 and 2 A general method for restoring degraded weather images is proposed. Based on the Denoising Diffusion Probabilistic Models (DDPM), the method restores degraded weather images through forward diffusion and backsampling. The specific process includes S1 to S6.

[0021] S1. The weather degradation image to be processed is segmented into multiple image blocks, and there is an overlapping area between two adjacent image blocks.

[0022] In S1, the specific methods for segmenting the weather degradation image to be processed into multiple image blocks include S11 to S13.

[0023] S11. Starting from one of the vertexes of the weather degradation image to be processed, determine a grid of size r×r. In this embodiment, it is preferable to use the upper left vertex of the weather degradation image as the starting point.

[0024] S12. Move the grid step by step with r as the action step size until the weather degradation image to be processed is decomposed into multiple grids.

[0025] S13. The weather degradation image to be processed is segmented based on a grid to form multiple image blocks. There is at least one overlapping grid between two adjacent image blocks. The size of the image block is denoted as c×c, and r<c.

[0026] Through S1, image blocks of the same size can be segmented from any weather degradation image to be processed, and then further processed, thereby effectively improving the generalization ability of the present invention.

[0027] S2. Noise blocks are obtained by forward diffusion, where noise is added to the image blocks multiple times. The method for adding noise to the image blocks multiple times is as follows: Where x0 is the image block, t is the forward time step, and α t Let ò be a predefined noise sequence, where ò is noise sampled from a standard normal distribution.

[0028] S3. The existing weather degradation image is used as a condition and input into the noise estimation network along with the inverse time step. The noise estimation network includes multiple basic DiT blocks and optimized DiT blocks that are stacked in an interleaved manner. The optimized DiT blocks are obtained by introducing cascaded context fusion attention based on the basic DiT blocks.

[0029] The basic DiT block mainly includes modules such as Multi-Head Self-Attention, Layer Normalization, Residual Connection, Pointwise Feedforward, and Adaptive layer norm (adaLN-Zero). Pointwise Feedforward enhances the model's representational power, helping it better distinguish different types of weather-degraded images and providing richer feature information for the restoration process. The adaLN-Zero module adaptively adjusts its parameters under different conditions, allowing the model to automatically adjust parameters according to varying weather conditions. adaLN-Zero uses a small neural network (typically an MLP) to conditionally generate a scaling factor γ, an offset factor β, and a dimension scaling parameter α for Layer Normalization based on the provided time-step embeddings. γ and β are applied to the block to achieve modulation, and α is applied before any residual link within the block. The structure and working principle of the basic DiT block are existing technologies in this field and will not be elaborated further here.

[0030] Because the multi-head self-attention blocks in the basic DiT block suffer from head redundancy, the model may be limited to recovering degraded images only under specific weather conditions, and cannot be applied to recovering degraded images across multiple weather conditions. Therefore, this invention employs Cascaded Contextual Fusion Attention (CCFA) to generate optimized DiT blocks. The method for generating optimized DiT blocks, based on the basic DiT block, includes: dividing the attention heads of the multi-head self-attention blocks in the basic DiT block into multiple groups, and distributing these groups sequentially; introducing a contextual attention mechanism between adjacent groups of attention heads; cascading these groups of attention heads based on the contextual attention mechanism to obtain a cascaded contextual fusion attention block; and replacing the multi-head self-attention blocks in the basic DiT block with the cascaded contextual fusion attention block to obtain the optimized DiT block. By optimizing the basic DiT block, the resulting optimized DiT block provides different segmentation features for each attention head, enabling the generation of diverse attention maps and enhancing the model's generalization performance when processing various degraded weather images.

[0031] S4. Generate estimated noise corresponding to the noise block using a noise estimation network, and calculate the average estimated noise based on the estimated noise of the overlapping region.

[0032] Please see Figure 3-5 The specific methods for generating estimated noise using a noise estimation network include S41 to S43.

[0033] S41. Convert the existing weather degradation images used as conditions into token vectors of quantity T and dimension D. The size of the token vector is N×T×D, where N is the batch size, D is the length of the token vector, and T = (H×W) / p 2 p is a hyperparameter, which is set to 2 in this embodiment, and H×W is the size of the noise block.

[0034] S42. Embed the reverse time step t' into all tokens.

[0035] S43. Process the Token one by one using the basic DiT block and the optimized DiT block to generate estimated noise.

[0036] The computation process of the cascaded context fusion attention block includes S431 to S434.

[0037] S431. Divide the input feature X in tensor form into H feature heads along the feature dimension: :Feats_in=split(X,H)∈R H×B×N×(D / H) ,in N is the batch size, T is the sequence length, and D is the feature dimension.

[0038] S432. Perform CFA and cascade operations on each feature head i: Feat i =CFA(Feat) i ).

[0039] The specific process of CFA calculation includes steps S4321 to S432.

[0040] S4321, Receive input data in tensor form Where B is the batch size, N is the sequence length, and C is the feature dimension.

[0041] S4322. Calculate Query(Q), Key matrix K1, and Value(V), with the results in matrix form:

[0042] S4323. Concatenate K1 with the input tensor X along the feature dimension:

[0043] S4324. Merge the spliced ​​matrix Y1 into the original dimension:

[0044] S4325. Calculate attention weights:

[0045] S4326. Calculate the new Key matrix K2:

[0046] S4327. Calculate the output matrix:

[0047] S433. Reassemble the processed feature headers: X_out=concat(Feat0,Feat1,...,Feat H-1 )∈R N×T×D .

[0048] S434. Merge all recombined feature headers: X_out = Fusion(X_out) ∈ R N×T×D .

[0049] CFA operations fuse key information with the input vector and incorporate contextual information when calculating attention weights, enabling the model to better capture the dependencies between local and global features and improve the accuracy of noise prediction for various weather-degraded images.

[0050] Specific methods for calculating the average estimated noise based on the estimated noise of the overlapping region include S435 to S437.

[0051] S435. Construct a matrix with the same size as the weather degradation image to be processed, and use the estimated noise corresponding to the pixels in the overlapping region as the element.

[0052] S436. Calculate the sum of all matrices to obtain...

[0053] S437. Based on the estimated noise level corresponding to each pixel in the overlapping region... Normalization is performed to obtain the average estimated noise.

[0054] S5. Smooth the average estimated noise and perform reverse implicit sampling on the noise block based on the smoothed average estimated noise.

[0055] S6. Reconstruct the restored blocks obtained by repeatedly performing reverse implicit sampling on the noise blocks to obtain the restored weather degradation image.

[0056] By merging all noise blocks based on average estimated noise, it is possible to input weather degradation images of any size and ensure high-quality restoration results, thus fully guaranteeing the generalization ability of this invention. Furthermore, by guiding the backsampling process through average estimated noise, smoothing is achieved between adjacent noise blocks, avoiding artifacts between two adjacent noise blocks after backsampling, and fully guaranteeing the clarity of the final restored image.

[0057] To verify the effectiveness of the present invention, the following experiment was conducted.

[0058] First, the Allweather dataset, composed of existing Raindrop, Outdoor-Rain, and Snow100K datasets, was used for training. Evaluation was performed on the synthetic test sets Raindrop-A, Test1, Snow100K-L, Snow100K-M, and Snow100K-S, and the real test sets RainDS and Snow100K-Real. The Allweather dataset contains 18,069 image pairs; the Raindrop dataset focuses on raindrop effects in images, particularly raindrops formed on surfaces such as windows; Raindrop contains 861 training image pairs and two test sets with 58 and 249 image pairs respectively. The Outdoor-Rain dataset contains both rain and fog, using a physics-based generative model to simulate dense synthetic raindrops. It contains 9000 image pairs, with a subset named Test1 (750 image pairs) used as the test set. Snow100K focuses on evaluating snow removal models. Its training set contains 50,000 image pairs, and its test set consists of three datasets: Snow100K-L, Snow100K-M, and Snow100K-S, representing different levels of snowfall. During testing, Raindrop-A (58 images) is used as the test set. For evaluation, RainDS and Snow100K-Real are used. RainDS contains 97 real test images, while Snow100K-Real contains 1329 real test images derived from real-world weather degradation.

[0059] Furthermore, image patches of 64×64 and 128×128 sizes were used, with sampling time steps of 10 and 25, respectively. In the 64×64 diffusion model training, the model sampled 16 images from the training set, and randomly cropped 16 64×64 image patches from each image. In the 128×128 diffusion model training, the model sampled 8 images from the training set, and randomly cropped 8 128×128 image patches from each image. The number of training iterations was set to 2,500,000. For the Raindrop-specific dataset, the Raindrop dataset was used for 2,000,000 training iterations. Additionally, the AdamW optimizer was used during training with a fixed learning rate of 0.00002 and no weight decay.

[0060] The datasets and training parameters described above are common knowledge in this field and will not be described in detail here.

[0061] After training, existing methods such as Uformer, Transweather, and WeatherDiff were compared and experimented with. The model generated by this invention is denoted as MDIR-UCDM, and the quantitative comparison is shown in Table 1, where the data in bold and underlined represent the results of the best-performing General Model and Integrated Model, respectively.

[0062] Table 1 Quantitative Comparison of Experimental Results

[0063] As can be seen from Table 1, the image restoration quality of this invention is superior to the comparative method. MDIR-UCDM 64 Compared to Transweather, it achieves a 0.99 dB improvement on the Snow100K-L test set, a 1.24 dB improvement on the Test1 dataset, and a 1.19 dB improvement on the Raindrop-A dataset. Uformer performs the worst on the OutdoorRain test set (Test1), indicating that existing integrated models cannot be easily applied to the restoration of multi-weather degraded images.

[0064] Furthermore, experiments were conducted using a single-scenario dataset, specifically the Raindrop dataset. The quantitative comparison results are shown in Table 2.

[0065] Table 2. Quantitative Comparison of Experimental Results for a Single-Scene Dataset

[0066] As shown in Table 2, this invention still achieves excellent performance compared to models specifically designed for raindrop removal. Specifically, compared to existing algorithms, this invention outperforms existing algorithms by an average of 0.14–4.55 dB on the Raindropo-A test set. Therefore, this invention can be used not only as a general model for multi-weather degraded image restoration but also as a specialized model for specific weather degraded image restoration, demonstrating good performance in both cases.

[0067] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A general method for restoring degraded images under various weather conditions, characterized in that, Includes the following steps: The weather degradation image to be processed is segmented into multiple image blocks, and there is an overlapping area between two adjacent image blocks; Noise blocks are obtained by forward diffusion of noise added to image blocks multiple times. Existing weather degradation images are used as conditions and fed into the noise estimation network along with the inverse time step. The noise estimation network consists of multiple basic DiT blocks and optimized DiT blocks that are stacked in an interleaved manner. The optimized DiT blocks are obtained by introducing cascaded context fusion attention based on the basic DiT blocks. A noise estimation network is used to generate estimated noise corresponding to the noise block, and the average estimated noise is calculated based on the estimated noise in the overlapping region. The average estimated noise is smoothed, and the noise block is implicitly sampled in reverse based on the smoothed average estimated noise. The restored weather degradation image is obtained by reconstructing the restored blocks obtained by repeatedly inverting implicit sampling of the noise blocks; The method for generating the optimized DiT block includes: The attention heads of the multi-head self-attention blocks in the basic DiT block are divided into multiple groups, and the multiple groups of attention heads are distributed sequentially. A contextual attention mechanism is introduced between two adjacent attention heads; multiple attention heads are concatenated based on the contextual attention mechanism to obtain a concatenated contextual fusion attention block; An optimized DiT block is obtained by replacing the multi-head self-attention block in the basic DiT block with a cascaded context fusion attention block.

2. The general multi-weather degraded image restoration method as described in claim 1, characterized in that, Specific methods for segmenting weather-degraded images into multiple image blocks include: Starting from one of the vertexes of the weather degradation image to be processed, determine a grid of size r×r; The grid is moved step by step with a step size of r until the weather degradation image to be processed is decomposed into multiple grids; The weather degradation image to be processed is segmented based on a grid, forming multiple image blocks, with at least one overlapping grid between two adjacent image blocks.

3. The general multi-weather degraded image restoration method as described in claim 1, characterized in that, The method for adding noise to image blocks multiple times is as follows: ； in, For image blocks, For positive time steps, Given a predefined noise sequence, It is noise sampled from a standard normal distribution.

4. The general multi-weather degraded image restoration method as described in claim 1, characterized in that, Specific methods for generating estimated noise using noise estimation networks include: Convert the existing weather degradation images, which are used as conditions, into a quantity of... And the dimension is The token vector, the size of the token vector is ,in For batch size, The length of the token vector. , For hyperparameters, The size of the noise block; Reverse time step Embedded into all tokens; The token is processed one by one using the basic DiT block and the optimized DiT block to generate estimated noise.

5. The general multi-weather degraded image restoration method as described in claim 1, characterized in that, Specific methods for calculating the average estimated noise based on the estimated noise in the overlapping region include: Construct a matrix of the same size as the weather degradation image to be processed, and use the estimated noise corresponding to the pixels in the overlapping region as the elements; Calculate the sum of all matrices to obtain ; Based on the estimated noise level corresponding to each pixel in the overlapping region... Normalization is performed to obtain the average estimated noise.

Images