A two-stage image restoration method for complex degradation

Abstract

This invention relates to the field of image processing technology, specifically to a two-stage image restoration method for composite degradation, comprising the following steps: S1: Inputting the degraded image into a restoration module to generate an intermediate image after degradation removal; the restoration module includes an encoder and a decoder, wherein the encoder uses a Transformer block to process cross-scale features in the spatial domain, and the decoder uses a WaveTrans block to restore image resolution and remove composite degradation in the frequency domain; S2: Inputting the intermediate image into a generation module to generate a clear image after detail reconstruction; the generation module introduces a signal thinning module and a frequency modulator to improve image quality. This invention effectively improves the effect of composite degradation image restoration by proposing a two-stage model.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and specifically to a two-stage image restoration method for complex degradation. Background Technology

[0002] Imaging under adverse weather conditions (such as rain, fog, and snow) leads to severe image degradation, posing a significant challenge to systems requiring all-weather operation, such as autonomous driving systems that rely on visual perception, outdoor monitoring equipment, and drone aerial photography. While significant progress has been made in dedicated and general models for single degradation scenarios, images in real-world scenarios are often simultaneously affected by multiple unpredictable and dynamically changing degradation factors. The complex interactions between these factors severely limit the performance of existing models. Figure 1 As shown, current state-of-the-art methods are inefficient and ineffective under conditions of combined degradation. Therefore, there is an urgent need to develop novel recovery frameworks to systematically address the problem of combined degradation.

[0003] Composite degradation is characterized by nonlinearity and randomness. It not only occludes local details and global structure in the spatial domain but also disrupts low-frequency and high-frequency components in the frequency domain, resulting in severe loss of original image information. In existing technologies, GUO et al. proposed the OneRestore framework in their journal paper "OneRestore: A Universal Restoration Framework for CompositeDegradation" (2024). While this framework addresses composite degradation by fusing degradation descriptors and image features through a cross-attention mechanism, it cannot simultaneously handle degradation removal and detail reconstruction, leading to residual degradation and smoothing phenomena in the output image (e.g., ...). Figure 1 (As shown).

[0004] Recently, the diffusion model proposed by HO et al. in their journal paper "Denoising Diffusion Probabilistic Models" (2020) has shown significant advantages in image restoration due to its rich prior texture details, generating high-quality images through a progressive denoising mechanism. However, the inherent randomness of its generation process and the lack of a conditional control mechanism make it difficult for general diffusion models to accurately preserve low-frequency structures, leading to structural distortion. To address this issue, ZHANG et al. proposed ControlNet in their journal paper "Adding Conditional Control to Text-to-Image Diffusion Models" (2023) to control the diffusion process of the image, ensuring the stability of the overall image structure. However, this model inevitably introduces noise during the process of injecting control signals into the denoising Unet through the encoder and intermediate block copies, which interferes with the denoising ability of the backbone network, resulting in abnormal details and artifacts in the generated image.

[0005] In summary, existing technologies still lack a complete solution that can collaboratively address spatial-frequency domain degradation interference, decouple and optimize degradation removal and detail reconstruction, while avoiding control signal noise pollution. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of the prior art and provide a two-stage image restoration method for complex degradation.

[0007] To achieve the above-mentioned objectives, the present invention provides the following technical solution:

[0008] A two-stage image restoration method for complex degradation includes the following steps:

[0009] S1: Input the degraded image into the restoration module to generate an intermediate image after removing the degradation; the restoration module includes an encoder and a decoder, wherein the encoder uses a Transformer block to process cross-scale features in the spatial domain, and the decoder uses a WaveTrans block to restore the image resolution and remove composite degradation in the frequency domain.

[0010] S2: Input the intermediate image into the generation module to generate a clear image with reconstructed details; the generation module introduces a signal thinning module and a frequency modulator to improve image quality.

[0011] Preferably, the encoder is composed of multiple Transformer blocks connected in series. The degraded image is input into the encoder, cross-scale features are extracted through the Transformer blocks, and the resolution is reduced by PixelUnshuffle to obtain the deepest features.

[0012] Furthermore, the decoder is composed of multiple WaveTrans blocks connected in series. Each WaveTrans block is composed of a Transformer block and a frequency learning module connected in series. The deepest feature is input into the Transformer block of the decoder to obtain intermediate features. Then, the degraded image and the intermediate features are input into the frequency learning module. The deepest feature is restored to its original size through the PixelShuffle operation to obtain the intermediate image after removing the degradation.

[0013] Furthermore, the frequency learning module processes the degraded image and intermediate features through three steps: frequency separation processing, mining processing, and thinning processing.

[0014] Furthermore, the frequency separation process adjusts the spatial size and channel dimension of the degraded image through 3×3 convolution to align it with the intermediate features, resulting in an aligned image. Then, discrete wavelet transform and 3×3 depthwise convolution are performed on the aligned image to obtain approximate coefficients, horizontal coefficients, vertical coefficients, and diagonal coefficients after local information enhancement. Zero-padding and discrete inverse wavelet transform are then applied to the approximate coefficients after local information enhancement to obtain denoised low-frequency information. Zero-padding and discrete inverse wavelet transform are then applied to the horizontal, vertical, and diagonal coefficients after local information enhancement to obtain high-frequency information containing texture details.

[0015] Furthermore, the mining process, guided by the low-frequency information and the high-frequency information, employs a Transformer cross-attention mechanism to mine low-frequency and high-frequency features from intermediate features; both the low-frequency features and the high-frequency features contain degradation components of the corresponding frequency band and original image information.

[0016] Furthermore, the thinning process uses a convolutional channel attention module and a convolutional spatial attention module to modulate the low-frequency features and the high-frequency features, respectively:

[0017] The convolutional channel attention module obtains channel-level attention by performing adaptive average pooling and adaptive max pooling on the low-frequency features, and suppresses redundant and degenerate features of the high-frequency features from the channel dimension to obtain the processed high-frequency features.

[0018] The convolutional spatial attention module obtains pixel-level attention by performing global average pooling and global max pooling on the high-frequency features, and suppresses the parts of the low-frequency features that are irrelevant to the image content along the spatial dimension to obtain the processed low-frequency features.

[0019] Then, a 1×1 convolution is used to aggregate the processed high-frequency features and the processed low-frequency features to generate modulated features; finally, the Transformer cross-attention mechanism is used to fuse the modulated features into the intermediate features to obtain the intermediate image.

[0020] Preferably, the generation module includes the following steps:

[0021] The intermediate image is input, and a content prompt is generated using the LLaVA large language model. A negative prompt is also introduced to suppress residual degradation of the intermediate image. This is then processed by the CLIP text editor into a prompt token ct. The prompt token ct, time step, and noise are input into the encoder of the denoising Unet, where they are frequency modulated by the frequency modulator of the denoising Unet jump connection section, and finally input into the decoder of the denoising Unet.

[0022] The intermediate image is input and processed into intermediate image features by a VAE encoder. The intermediate image features and noise are input into the encoder and intermediate block copy, then the features are optimized by the signal refinement module, and finally input into the decoder part of the denoising Unet.

[0023] The denoised image features are output from the decoder portion of the denoised Unet and then used by the VAE decoder to generate a clear image.

[0024] Furthermore, a signal refinement module is introduced at the output of the encoder and the intermediate block copy, and a frequency modulator is introduced at the jump connection part of the denoising Unet to remove noise;

[0025] The signal thinning module adjusts the scaling factor according to the adaptive characteristics of specific features, thereby dynamically adjusting the scaling ratio of each feature map and limiting the scaling operation to half of the channels.

[0026] The frequency modulator uses Fourier transform for spectral modulation to reduce low-frequency components in the denoising Unet jump connection portion.

[0027] Preferably, the recovery module adopts the MSE loss function optimization model:

[0028]

[0029] in The MSE loss function for the recovery module, Y This is an intermediate image. X For the input image, Represents the L2 norm;

[0030] The generation module optimizes the model by minimizing the following potential diffusion objectives:

[0031]

[0032] in Optimize the model loss function to generate the potential diffusion target of the module. For added noise, For text prompts, For time steps, The features of the intermediate image after passing through the VAE encoder. Gaussian noise was added to the image to represent reality. For noise predicted by the condition generator, Let represent the expectation over the sampling space of all variables, and represent the average loss during sampling training. This represents the L2 norm.

[0033] Compared with the prior art, the present invention has the following beneficial effects:

[0034] This invention proposes a two-stage image restoration method for composite degradation, dividing image restoration modeling into two stages: degradation removal and detail reconstruction. In the degradation removal stage, the restoration module CDRNet (CompositeDegradation Removal Net) collaboratively removes degradation from both the spatial and frequency domains. In the detail reconstruction stage, the signal thinning module and frequency modulator of the generation module FreeControlNet significantly improve the generation quality without increasing computational cost. Due to CDRNet's cross-scale information modeling capabilities and FreeControlNet's rich detail texture priors, this invention effectively improves the performance of composite degradation image restoration compared to current methods. Attached Figure Description

[0035] Figure 1 This is a schematic diagram illustrating the processing effect of the current state-of-the-art method in the background technology on the restoration of rain and snow composite degraded images;

[0036] Figure 2 This is a simplified flowchart of the two-stage image restoration method for compound degradation in Example 1;

[0037] Figure 3 This is a flowchart illustrating the two-stage image restoration method for composite degradation in Example 1.

[0038] Figure 4 This is a schematic diagram of the internal structure of the recovery module in Example 1;

[0039] Figure 5 Here is a simplified flowchart of the frequency learning module in Example 1;

[0040] Figure 6 This is a schematic diagram of the Transformer cross-attention mechanism structure in Example 1;

[0041] Figure 7 Here is a simplified flowchart of the generation module in Example 1;

[0042] Figure 8 This is a schematic diagram illustrating the processing effects of the present invention and existing technology on the restoration of composite degraded images in Example 2. Detailed Implementation

[0043] The present invention will now be described in further detail with reference to specific embodiments. However, this should not be construed as limiting the scope of the present invention to the following embodiments; all technologies implemented based on the content of the present invention fall within the scope of the present invention.

[0044] Unless otherwise specified, the use of terms such as "upper," "lower," "left," "right," "center," "inner," and "outer" to indicate orientation or positional relationships in the description of specific embodiments of the present invention is based on the orientation or positional relationships shown in the accompanying drawings, or the orientation or positional relationship in which the product / equipment / device is typically placed during use. These terms are merely for the purpose of facilitating the description of the present invention or simplifying the description in specific embodiments, enabling those skilled in the art to quickly understand the solution, and do not indicate or imply that a particular device / component / element must have a specific orientation, or be constructed and operated in a specific positional relationship. Therefore, they should not be construed as limitations on the present invention.

[0045] Furthermore, the use of terms such as "horizontal," "vertical," "suspended," and "parallel" does not imply that the corresponding device / component / element must be absolutely horizontal, vertical, suspended, or parallel, but rather that it can be slightly tilted or have a deviation. For example, "horizontal" merely means that its direction is more horizontal relative to "vertical," not that the structure must be completely horizontal, but that it can be slightly tilted. Alternatively, it can be simplified to mean that the corresponding device / component / element, when set in a "horizontal," "vertical," "suspended," or "parallel" direction, can have an error / deviation of ±10% relative to the corresponding direction, more preferably within ±8%, more preferably within ±6%, more preferably within ±5%, and more preferably within ±4%. As long as the corresponding device / component / element is within the error / deviation range, it can still achieve its function in the present invention.

[0046] Furthermore, the use of terms such as "first," "second," and "third" in terminology is merely for distinguishing descriptions of identical or similar components and should not be interpreted as emphasizing or implying the relative importance of a particular component.

[0047] Furthermore, in the description of the embodiments of the present invention, "several", "more than", and "a number of" represent at least two. The number can be any number, such as 2, 3, 4, 5, 6, 7, 8, or 9, and can even exceed nine.

[0048] Furthermore, in the description of the technical solution of this invention, unless otherwise explicitly specified / limited / restricted, the terms "set up," "install," "connect," "link," "provided with," "laid out," and "arranged" should be interpreted broadly. For example, they can refer to fixed connections, detachable connections, or integral connections; they can refer to common connection methods in the art, such as welding, riveting, bolting, and threaded connections. Such connections can be mechanical, electrical, or communication connections; they can be direct connections or indirect connections through an intermediate medium; and they can refer to the internal communication between two components.

[0049] Example 1

[0050] Figure 2 and Figure 3 A flowchart of a two-stage image restoration method for compound degradation is presented, which decomposes the compound degradation restoration task into a two-stage process: a restoration module that removes image-irrelevant degradations using the restoration module CDRNet, and a generation module FreeControlNet that regenerates image details. These two stages are decoupled and optimized independently.

[0051] In the first stage, degraded images X The degradation is removed by processing with the CDRNet restoration module to generate an intermediate image. Y .like Figure 3 As shown in (a), CDRNet comprises a four-layer encoder and a four-layer decoder. During downsampling, this invention uses Transformer blocks to process cross-scale features in the spatial domain; during upsampling, WaveTrans blocks are used to restore image resolution and remove composite degradation in the frequency domain.

[0052] In the second stage, intermediate images Y The FreeControlNet module is used to process the image to reconstruct details and generate a sharp image. Z .like Figure 3 As shown in (b), the generation module includes the following steps:

[0053] Input intermediate image YThe LLaVA large language model is used to generate content cues to control the overall image structure during the denoising process. To avoid incomplete degradation removal in the first stage from contaminating subsequent processes, negative cues (such as rain, fog, and snow) are introduced to suppress residual degradation in the intermediate image Y. The image is then processed by the CLIP text editor into a prompt token ct. The prompt token ct, time step t, and noise are then combined. The encoder part of the input denoising Unet is frequency modulated by the frequency modulator of the jump connection part of the denoising Unet, and finally input to the decoder part of the denoising Unet.

[0054] Input intermediate image Y The image features are processed by a VAE encoder. ; to extract intermediate image features and noise The input encoder and intermediate block copy are then processed by a signal refinement module to optimize the features, and finally input into the decoder part of the denoising Unet.

[0055] The decoder part of the denoising Unet outputs the denoised image features. A clear image Z is generated by the VAE decoder.

[0056] Considering the cross-scale characteristics of composite degradation in both the spatial and frequency domains, this invention proposes CDRNet, a Transformer-based encoder-decoder architecture designed to process input degraded images... X Remove composite degradation and output intermediate image Y . Figure 4 This is a schematic diagram of the internal structure of the recovery module. Figure 4 Figure (a) illustrates the internal structure of the Transformer block in each layer of the encoder. The input degraded image first undergoes layer normalization and Transposed Space Attention (TSA) processing sequentially, followed by a residual operation. Subsequently, it passes through layer normalization and a FeedForward Network (FFN) sequentially, and finally undergoes a second residual operation. The TSA and FFN utilize Restormer's GDFN (Gated-Dconv Feed-Forward Network) and MDTA (Multi-Dconv HeadTransposed Attention). Figure 4 Figure (b) describes the internal structure of the WaveTrans block in each layer of the decoder, which consists of a wavelet transform-based Frequency Learning Module (FLM). Figure 4(as shown in (c)) is formed by concatenation after the Transformer block; Figure 4 (d) and Figure 4 In the diagram, (e) represents the Convolutional Spatial Attention (CSA) module and the Convolutional Channel Attention (CCA) module, respectively; DWT (Discrete Wavelet Transform) and IDWT (Inverse Discrete Wavelet Transform) refer to the Discrete Wavelet Transform and its inverse transform, respectively, while TCA (Transformer Cross-Attention) represents Transformer Cross-Attention.

[0057] Specifically, given a size of H × W A 3×3 degraded image (representing an image with height H, width W, and 3 channels) is used to generate the deepest features through a four-layer encoder. During this process, the Transformer block leverages its powerful ability to model global and local features, extracting cross-scale features at each layer. Downsampling is performed using PixelUnshuffle, progressively reducing resolution by increasing the number of channels.

[0058] Next, the deepest feature is input into the four-layer decoder, and the deepest feature is restored to its original size through a PixelShuffle operation. During this process, the deepest feature is input into the Transformer block within the WaveTrans block to obtain intermediate features. F Then degraded image X and intermediate features F The input frequency learning module, with its powerful cross-scale frequency analysis capabilities, first processes the degraded image... X Decomposed into different frequency subbands, and then TCA was used to extract intermediate features. F High and low frequency image content is extracted, with each frequency band containing its own degradation components and original image information. Finally, the features are refined through a bidirectional convolutional attention module, which either suppresses or allows content of specific frequencies to pass through, thereby enhancing components unrelated to degradation and achieving the removal of composite degradation.

[0059] like Figure 4 (c) and Figure 5 As shown, the degraded image X and intermediate features ( (H represents the set of real numbers, W represents the height, and C represents the number of channels.) Simultaneously, the input to the frequency learning module is used to process the degraded image through discrete wavelet transform and its inverse transform. X Perform spectral decoupling, from intermediate features F The frequency learning module proposed in this invention mainly consists of three parts: frequency separation, mining, and refinement.

[0060] For the frequency separation part, the degraded image X First, the spatial dimensions and channel dimensions of the convolutional group are compared with the intermediate features. F Alignment yields the aligned image. The two-dimensional wavelet coefficients are then calculated using Discrete Wavelet Transform (DWT):

[0061]

[0062] in, For the aligned image, , , and These represent the aligned images. The approximation coefficients, horizontal coefficients, vertical coefficients, and diagonal coefficients, This represents the discrete wavelet transform. Each coefficient is processed as follows:

[0063]

[0064] in, , , and These represent the aligned images. The approximation coefficients, horizontal coefficients, vertical coefficients, and diagonal coefficients, , , , All of these represent depthwise convolutions with a kernel size of 3×3. The subscripts (a, h, v, d) indicate that the four coefficients are processed by four independent convolutions to obtain the approximate coefficients LL, LH, HL, and HH of the local information enhancement.

[0065] To process low-frequency and high-frequency information separately, this invention employs zero-padding and discrete inverse wavelet transform (IDWT) to obtain the denoised low-frequency information (in... Figure 4 The upper branch of (c) and Figure 5 (represented in the upper branch) and high-frequency information containing texture details (in) Figure 4 The lower branch of (c) and Figure 5 (This is represented in the next branch). The specific processing steps are as follows:

[0066]

[0067]

[0068] in, , , and These represent the approximation coefficient, horizontal coefficient, vertical coefficient, and diagonal coefficient of local information enhancement, respectively. and These refer to the degraded images respectively. X The low-frequency and high-frequency information separated from it Represents the discrete inverse wavelet transform. This indicates zero padding, and concat(.) indicates concatenation.

[0069] The next part is the excavation section, which is where this invention... and Guided by TCA, it uses intermediate features F Mining different feature components and extracting low-frequency features and high frequency characteristics It contains the degradation components of the corresponding frequency band and the original image information.

[0070] The following section details the refinement process, where the invention uses two convolutional attention units, CCA and CSA, to modulate low-frequency features. and high frequency characteristics This reduces the impact of degradation factors in each frequency band and integrates image content. For CCA, this invention reduces the impact of low-frequency features. Adaptive average pooling and adaptive max pooling are used to obtain channel-level attention. Suppressing high-frequency features from the channel dimension The redundancy and degradation characteristics in [the system / process] are described below.

[0071]

[0072]

[0073] in, It is a high-frequency feature. These are the processed high-frequency features. These are the channel-level features after CCA processing. Represents element-wise multiplication. (i=1, 2, 3, 4) represents a 1×1 convolution, where the subscript i indicates the index for differentiation purposes only. δ is the Sigmoid function, γ is the ReLU activation function, and AAP(.) and AMP(.) represent adaptive average pooling and adaptive max pooling, respectively. It is a low-frequency characteristic.

[0074] For CSA, by analyzing high-frequency features Pixel-level attention is obtained by performing global average pooling and global max pooling. Suppress low-frequency features that are irrelevant to image content along the spatial dimension. The operation process is described as follows:

[0075]

[0076]

[0077] in, It is the processed low-frequency feature. It is a low-frequency characteristic. These are pixel-level features after CSA processing. Represents element-wise multiplication. (.) represents a 7×7 convolution, δ is the sigmoid function, concat(.) represents concatenation, and GAP(.) and GMP(.) represent global average pooling and global max pooling, respectively. It is a high-frequency feature.

[0078] Finally, we use 1×1 convolution to aggregate the processed low-frequency features. and processed high-frequency features Generate modulated features Then, TCA is used to modulate the features. Integrating into intermediate features F In the process, an intermediate image is obtained. Y By progressively processing features at different resolutions across frequency scales during the upsampling process, effective removal of composite degradation is achieved.

[0079] exist and Under the guidance of this invention, TCA (such as...) is applied. Figure 6 (As shown) From input intermediate features F Different feature components are extracted from the data. The specific processing steps are as follows:

[0080]

[0081]

[0082]

[0083]

[0084] in It is a low-frequency / high-frequency indicator. Indicates low-frequency features / high-frequency features. This represents low-frequency / high-frequency information, where T is the transpose operation and softmax() is the softmax operation. , V and V represent the query, key, and value projections, respectively. It is a scaling factor used to control and The size of the dot product. (where i = 1, 2, 3) represents a 3×3 depthwise convolution, and the subscript i represents the index, which is only for the purpose of differentiation; (where j=5, 6, 7) represents a 1×1 pointwise convolution, and the subscript j represents the index, which is only for the purpose of differentiation; F It is an intermediate feature.

[0085] like Figure 3 (b) and Figure 7 As shown, since the encoder and intermediate block copy of the generation module FreeControlNet introduce noise into the backbone of the denoising Unet, thereby interfering with the denoising process and affecting image quality, the present invention introduces a signal refinement module at the output end of the encoder and intermediate block copy, and introduces a frequency modulator in the jump connection part of the denoising Unet.

[0086] For the output of the encoder and intermediate block copies, this invention dynamically adjusts the scaling ratio of each feature map. Unlike applying a fixed scaling factor uniformly to all samples or positions within the same channel, the method of this invention adjusts the scaling factor based on the adaptive characteristics of specific features. First, this invention calculates the average feature map along the channel dimension:

[0087]

[0088] in Represents the average feature map. Indicates the control signal feature map at position The first The value of each channel, This represents the total number of channels. The scaling factor plot is then calculated as follows:

[0089]

[0090] In the formula This represents the scaling factor map, where S is a constant used to control the scaling ratio. , , These represent the average feature map, minimum average feature map, and maximum average feature map, respectively. The present invention then experimentally discovered that indiscriminately amplifying all channels of the control signal feature map would excessively amplify the low-frequency control signal, resulting in a loss of texture detail. Therefore, the present invention restricts the scaling operation to half of the channels.

[0091]

[0092] in This represents the control signal feature map after limiting the scaling operation to half of the channels. Indicates the control signal feature map at position The first The value of each channel, Represents the total number of channels. Represents the scaling factor plot. This represents element-wise multiplication.

[0093] To further mitigate the texture smoothing problem caused by enhanced low-frequency control, this invention utilizes Fourier transform for spectral modulation to reduce low-frequency components in skip connections, as shown below:

[0094]

[0095]

[0096]

[0097]

[0098] In the formula This represents the feature map before modulation after Fourier transform. This indicates that the frequency intensity scaling process is applied to the pre-modulation feature map after Fourier transform. and These are feature maps before and after modulation. For modulation factor, This is a modulation mask used to achieve frequency intensity scaling, where r is the radius. For threshold frequency, FFT(.) represents element-wise multiplication, and FFT(.) and IFFT(.) represent the Fourier transform and its inverse operation.

[0099] It is worth noting that the present invention only applies the signal refinement module and frequency modulator during inference, and this method can significantly improve image quality without additional training and fine-tuning.

[0100] For the recovery module, this invention uses MSE loss for optimization:

[0101]

[0102] in The MSE loss function for the recovery module, Y This is an intermediate image. X For the input image, This represents the L2 norm.

[0103] For the generation module, the objective of this invention is to minimize the following potential diffusion targets:

[0104]

[0105] in Optimize the model loss function to generate the potential diffusion target of the module. For added noise, For text prompts, For time steps, The features of the intermediate image after passing through the VAE encoder. Gaussian noise was added to the image to represent reality. For noise predicted by a condition generator (such as Unet), Let represent the expectation over the sampling space of all variables, and represent the average loss during sampling training. This represents the L2 norm.

[0106] Example 2

[0107] To more comprehensively simulate complex degradation phenomena in the real world, this invention uses the LCDD dataset, which contains three composite degradation types: rain-fog, rain-snow, and snow-fog. LCDD contains 26,252 training image pairs and 37,952 test image pairs. For snow-fog and snow-rain degradation, the dataset includes 9,001 training pairs and 16,801 test pairs, respectively. For rain-fog degradation, the dataset contains 8,250 training pairs and 750 test pairs. Since simultaneous occurrences of rain, snow, and fog are rare, they are not considered within the scope of this study. All ground truth (GT) images are taken from the All-weather dataset.

[0108] This invention trains the recovery module for 150k iterations (batch size = 24). Then, StableDiffusion 2.1-base is used as the generation prior, and the proposed FreeControlNet is fine-tuned for 100k iterations (batch size = 128). The optimizer used is Adam. The learning rate is set to a certain value in the first 40k iterations. Subsequently, in the next 60k iterations, it was reduced to The training process was conducted at a resolution of 512×512.

[0109] This embodiment compares the present invention with three one-to-one image restoration methods (MPRNet, NAFNet, Restormer), eight one-to-many image restoration methods (AirNet, Transweather, WeatherDiff, DiffIR, X-restormer, Histoformer, DiffUIR, AdaIR), and one one-to-one composite method (OneRestore). Table 1 shows a quantitative comparison of the present invention with state-of-the-art methods on the LCDD dataset, demonstrating that the present invention achieves near-optimal performance across all tasks. Compared to OneRestore, which handles composite degradation, the present invention achieves significant improvements in removing rain-fog, snow-fog, and snow-rain degradation, with PSNR improvements of 2.28 dB, 3.32 dB, and 2.96 dB, respectively. Furthermore, compared to AdaIR (ICLR'25), the present invention improves the average PSNR by 0.65 dB. Figure 8 As shown in (a), the present invention completely removes snow from the ground; as Figure 8 (b) and Figure 8 As shown in (c), it restores details on windows and roofs more closely to the ground truth. This invention outperforms OneRestore in removing degradation, performing more thoroughly, and is more accurate in reconstructing texture details.

[0110] To demonstrate the generalization ability of this invention, its performance was tested on another composite degradation dataset—CDD (only composite degradation scenarios were tested; single degradation scenarios are not within the scope of this study). As shown in Table 2, this invention exhibits stronger generalization ability than AdaIR. Furthermore, as... Figure 8 As shown in (d), the present invention produces a more natural transition at the edge of an object without introducing artifacts.

[0111] Table 1 Comparison of quantitative results on the LCDD dataset

[0112]

[0113] Table 2 Quantitative comparison of generalization performance

[0114]

[0115] The above quantitative and qualitative comparison results demonstrate the superiority of the method of the present invention over other methods in complex degraded image restoration tasks.

[0116] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A two-stage image restoration method against composite degradation, characterized in that, The method comprises the following steps: S1: input the degraded image into the recovery module to generate an intermediate image after removing the degradation; the recovery module comprises an encoder and a decoder, wherein the encoder adopts a Transformer block to process cross-scale features in a spatial domain, and the decoder adopts a WaveTrans block to restore the image resolution and remove the composite degradation in a frequency domain; The encoder is composed of multiple layers of Transformer blocks connected in series, the degraded image is input into the encoder, cross-scale features are extracted through the Transformer blocks, and the resolution is reduced through a PixelUnshuffle operation to obtain the deepest layer features; The decoder is composed of multiple layers of WaveTrans blocks connected in series, a WaveTrans block is composed of a Transformer block and a frequency learning module connected in series, the deepest layer features are input into the Transformer block of the decoder to obtain intermediate features, and then the degraded image and the intermediate features are input into the frequency learning module, the deepest layer features are restored to the original size through a PixelShuffle operation to obtain the intermediate image after removing the degradation; S2: input the intermediate image into the generation module to generate a clear image after detail reconstruction; the generation module introduces a signal refinement module and a frequency modulator to improve the image quality; The generation module comprises the following steps: input the intermediate image, generate the content prompt of the image through the LLaVA large language model; at the same time, introduce a negative prompt to suppress the residual degradation of the intermediate image; then process it into a prompt token ct through a CLIP text editor; input the prompt token ct, time step and noise into the encoder part of the denoising Unet, and perform frequency modulation through the frequency modulator of the skip connection part of the denoising Unet; finally, input into the decoder part of the denoising Unet; input the intermediate image, process it into an intermediate image feature through a VAE encoder; input the intermediate image feature and noise into the encoder and the intermediate block copy, then optimize the feature through the signal refinement module, and finally input into the decoder part of the denoising Unet; output the denoised image feature from the decoder part of the denoising Unet, and generate a clear image through a VAE decoder.

2. The two-stage image restoration method against composite degradation according to claim 1, wherein, The frequency learning module processes the degraded image and the intermediate feature through three steps of frequency separation processing, mining processing and refinement processing.

3. The two-stage image restoration method against composite degradation according to claim 2, characterized in that, The frequency separation processing adjusts the spatial size and channel dimension of the degraded image through a 3x3 convolution to align it with the intermediate feature, and obtains an aligned image; then performs discrete wavelet transform and 3x3 depth-wise convolution on the aligned image to obtain approximate coefficients, horizontal coefficients, vertical coefficients and diagonal coefficients after local information enhancement; then uses zero padding and discrete inverse wavelet transform to obtain low-frequency information after denoising on the approximate coefficients after local information enhancement; uses zero padding and discrete inverse wavelet transform on the horizontal coefficients, vertical coefficients and diagonal coefficients after local information enhancement to obtain high-frequency information containing texture details.

4. The two-stage image restoration method against composite degradation according to claim 3, characterized in that, The mining processing uses a Transformer cross-attention mechanism to mine low-frequency features and high-frequency features from the intermediate features through the guidance of the low-frequency information and the high-frequency information. The low-frequency features and the high-frequency features both contain the degradation components and the original image information of the corresponding frequency bands.

5. The two-stage image restoration method against composite degradation according to claim 4, wherein, The refinement processing uses a convolution channel attention module and a convolution spatial attention module to modulate the low-frequency features and the high-frequency features, respectively: The convolution channel attention module obtains channel-level attention by performing adaptive average pooling and adaptive maximum pooling on the low-frequency features, suppresses redundant degradation features of the high-frequency features in the channel dimension, and obtains processed high-frequency features; The convolution spatial attention module obtains pixel-level attention by performing global average pooling and global maximum pooling on the high-frequency features, suppresses the parts of the low-frequency features that are irrelevant to the image content along the spatial dimension, and obtains processed low-frequency features; Then, the processed high-frequency features and the processed low-frequency features are aggregated using a 1×1 convolution to generate modulated features; finally, the modulated features are fused into the intermediate features using a Transformer cross-attention mechanism to obtain an intermediate image.

6. The two-stage image restoration method against composite degradation according to claim 1, wherein, A signal refinement module is introduced at the output end of the encoder and the intermediate block copy, and a frequency modulator is introduced at the skip connection part of the denoising Unet to remove noise; The signal refinement module adjusts the scaling factor according to the adaptive characteristics of specific features, thereby dynamically adjusting the scaling ratio of each feature map, and limiting the scaling operation within half of the channels; The frequency modulator uses Fourier transform for spectral modulation to reduce the low-frequency components of the denoising Unet skip connection part.

7. The two-stage image restoration method against composite degradation according to claim 1, wherein, The restoration module optimizes the model using an MSE loss function: wherein is the MSE loss function of the restoration module, Y is the intermediate image, X is the input image, denotes the L2 norm; The generation module optimizes the model by minimizing the following latent diffusion objective: wherein is a potential diffusion target optimization model loss function for the generation module, is added noise, is a text prompt, is a time step, is a feature of the intermediate image after the VAE encoder, is real added Gaussian noise, is noise predicted by the conditional generator, denotes expectation over the sampling space of all variables, representing the average loss of the sampled training, denotes L2 norm.

Images