Image restoration method based on enhanced conditional diffusion model of dual-domain interaction Transformer

Abstract

The application discloses an image restoration method based on a double-domain interaction Transformer enhanced condition diffusion model, which comprises the following steps: diffusion training, that is, forward noise adding and reverse noise removing are performed on a clean image x0, and a degraded image is used as a condition to guide in the reverse noise removing process; enhancement training, that is, Gaussian noise is removed by using a noise estimation network, a clean image is finally obtained, and a target function is redesigned to further optimize parameters θ; and image restoration is performed on a degraded image to be restored by using the trained noise estimation network. The method can significantly improve the image restoration performance.

Description

Technical Field

[0001] This invention belongs to the field of image restoration, specifically, it relates to an image restoration method based on an enhanced conditional diffusion model of a dual-domain interactive Transformer. Background Technology

[0002] Image restoration aims to reconstruct clean images by removing various types of degradation, playing a crucial role in multiple computer vision fields such as autonomous driving and remote sensing image processing. The key to image restoration is removing noise and distortion without introducing additional artifacts. Therefore, image restoration is essential for ensuring performance on high-level vision tasks. Traditional image restoration methods rely on hand-defined prior knowledge and cannot handle complex degradation. Most existing state-of-the-art image restoration methods are based on deep learning, including methods based on Convolutional Neural Networks (CNNs) and Transformers. The performance of image restoration has been significantly improved due to the powerful feature representation capabilities of neural networks and the emergence of large-scale pairwise datasets. Recently, diffusion models have been introduced into image processing, providing a new perspective and approach for image restoration. The inverse process of a diffusion model can be viewed as a denoising task; accurate noise estimation enables the diffusion model to progressively generate rich details. However, noise estimation in diffusion-based restoration methods relies on U-net, which may not fully capture the nuances required for noise estimation, thus affecting the performance of image restoration. Furthermore, most diffusion-based methods primarily focus on predicting Gaussian noise in the original pixel space, limiting the exploration of image frequency features. Therefore, there is still significant room for improvement in the noise estimation networks of existing diffusion models.

[0003] To address these issues, the performance of diffusion models in image restoration can be enhanced by improving the accuracy of noise estimation networks. Frequency analysis has received considerable attention in computer vision, but most methods rely on only a single frequency analysis technique, such as Discrete Wavelet Transform (DWT) or Fast Fourier Transform (FFT). DWT provides a localized frequency representation by transforming the signal into multiple frequency subbands. This allows models to analyze signals at multiple scale levels, distinguishing between high-frequency noise components and low-frequency signal components. However, DWT lacks the finesse of frequency resolution found in FFT, making it less capable of selecting frequency components rich in information. FFT's noise spectrum provides an efficient way to analyze global frequency content, but it lacks multi-scale decomposition capabilities and fails to provide an efficient sparse representation similar to DWT. Summary of the Invention

[0004] In view of this, the purpose of this invention is to provide an image restoration method based on a dual-domain interactive Transformer enhanced conditional diffusion model, the method comprising:

[0005] Step 1: Perform diffusion training, which involves forward noise addition and reverse denoising on the clean image x0, and using the degraded image during the reverse denoising process. As a conditional guide, a noise estimation network with parameter θ is used in the reverse denoising process. At time step t, the mean of the noise estimation network is... in, This represents the sampled image during the reverse denoising process at time step t.

[0006] During diffusion training, a loss function is used. As the objective function, through continuous training, it becomes... Minimum, of which in, This represents calculating the mean, ||||1 represents the L1 norm, and ε t This represents the noise added during the positive noise addition at time step t. This represents the noise estimated by the noise estimation network at time step t;

[0007] Step 2: Perform reinforcement training, i.e., Gaussian noise A noise estimation network is used for denoising to obtain a clean image. Using degraded images during denoising Use it as a condition for guidance, and set the loss function As the objective function for the enhanced training phase, the parameter θ is further optimized.

[0008] in,

[0009]

[0010]

[0011] y represents the degraded image. The true clean image, SSIM(x′,y′) represents the structural similarity between any two images x′ and y′, μ x and μ y Let x and y represent the means of the images, respectively. and Let σ represent the variances of images x′ and y′, respectively. xy The covariance of images x′ and y′ is represented by C1 and C2, which are constants used to prevent instability when the denominator approaches zero. α is a hyperparameter. This is the clean image obtained from the final sampling;

[0012] Step 3: Using the trained noise estimation network, perform image restoration on the degraded image to be restored.

[0013] The noise estimation network is a dual-domain interactive transformer structure, which includes a 3×3 first convolutional layer, an MFSIM module, seven stacked DDITB modules, and a 3×3 second convolutional layer in sequence according to the signal input direction. The MFSIM module is used to obtain multi-scale frequency domain and spatial domain fusion features. The output of the first DDITB module is concatenated with the output of the sixth DDITB module as the input of the seventh DDITB module, the output of the second DDITB module is concatenated with the output of the fifth DDITB module as the input of the sixth DDITB module, and the output of the third DDITB module is concatenated with the output of the fourth DDITB module as the input of the fifth DDITB module.

[0014] Furthermore, a variance sequence {β1,β2,…,β} is applied to the clean image x0. T Gaussian noise is added forward to generate a noisy image x after T time steps. T Noisy image x T Approximate Gaussian noise, the process of positive noise addition can be expressed by the following formula:

[0015]

[0016]

[0017] Where q represents the conditional probability distribution and t represents the time step. Let I represent a Gaussian distribution, and let x represent the identity matrix. t This represents the image at time step t.

[0018] Furthermore, the reverse denoising process is based on Start learning Gaussian denoising transformation and using degraded images. As a condition, the reverse denoising process is expressed by the following formula:

[0019]

[0020]

[0021] in, α t =1-β t , This represents the sampled image at time step t. This represents the noise variance at time step t. This represents the mean of the noise estimation network at time step t. This represents the noise estimated by the noise estimation network at time step t.

[0022] Furthermore, the MFSIM module includes a frequency domain unit and a spatial unit. In the frequency domain unit, wavelet transform is used to obtain multi-scale frequency domain features. The spatial unit includes three branches, which use convolution kernels of sizes 3×3, 5×5 and 7×7 to extract multi-scale spatial features. The output of the frequency domain unit is used as the input of the spatial unit, thereby realizing the interaction and fusion of features in the frequency domain and the spatial domain.

[0023] Furthermore, the DDITB module includes a dual-domain self-attention unit and a frequency-domain learnable feedforward network. The DDITB module processes the input features F. in Layer normalization is performed, and sinusoidal positional encoding is used to convert time step t into temporal embedding T′. The input feature F after layer normalization is then processed. in The input is added to the temporal embedding T′ and fed into a dual-domain self-attention unit. The dual-domain self-attention unit performs attention calculations on the input in both the frequency and spatial domains, yielding its output. After layer regularization, the output of the dual-domain self-attention unit, along with the temporal embedding T′, is then input into a frequency-domain learnable feedforward network.

[0024] The calculation process of the DDITB module can be expressed by the following formula:

[0025] F in =DDSA(LN(F) in )+T′)+F in ,

[0026] F out =FLFFN(LN(F in ′)+T′)+F in ′,

[0027] Among them, F in LN(·) represents the input to the DDITB module, LN(·) represents layer normalization, DDSA(·) represents the operation of the dual-domain self-attention unit, FLFFN(·) represents the operation of the frequency-domain learnable feedforward network, and F in ′ represents the output of the two-domain self-attention unit, F out This indicates the output of the DDITB module.

[0028] Furthermore, the dual-domain self-attention unit performs attention calculations on the input in both the frequency and spatial domains, specifically including:

[0029] Input features X of the two-domain self-attention unit in Temporal embedding is used to generate three matrices: query Q, key K, and value V, by applying 1×1 pointwise convolution and 3×3 depthwise convolution.

[0030] The calculation process of dual-domain self-attention is expressed by the following formula:

[0031] DDSA = Concat[DDSA] f (Q,K,V),DDSA s (Q,K,V)]+X in ,

[0032] in,

[0033]

[0034]

[0035]

[0036]

[0037]

[0038] DDSA f (·) and DDSA s (·) represents the DDSA operation in the frequency domain and spatial domain, respectively; Concat[·] represents the concatenation operation; W p and W d These represent a 1×1 pointwise convolution kernel and a 3×3 depthwise convolution kernel, respectively, with W superscripted as Q, K, and V. p and W d In the equations, represent the weights when calculating query Q, key K, and value V, respectively, and β represents the learnable scaling parameter. This represents the Fast Fourier Transform (FFT). This represents the Inverse Fast Fourier Transform (IFFT). This represents the conjugate transpose of the Fast Fourier Transform.

[0039] Furthermore, the frequency-domain learnable feedforward network performs the following operations:

[0040] Enter X i ′ n After layer normalization and temporal embedding, it is divided into X. f and X s , that is, X f ,X s =Split(LN(X) i ′ n )+T′), Split(·) represents the splitting operation;

[0041] X fThe input is fed into a linear layer and transformed to the frequency domain using FFT. The frequency domain result is then multiplied element-wise with the learnable weights W, followed by IFFT to transform it back to the spatial domain. Finally, it is activated by GELU and processed through a linear layer to generate X′. f ,Right now This represents the Fast Fourier Transform (FFT). Represents Inverse Fast Fourier Transform (IFFT), Conv 1×1 This represents a 1×1 convolution in a linear layer;

[0042] X s X is generated after passing through a linear layer, GELU activation, and another linear layer in sequence. s ′, i.e., X s ′=Conv 1×1 (GELU(Conv 1×1 (X s )));

[0043] X′ f and X s Connect the '' and add the input X. i ′ n The output of the frequency-domain learnable feedforward network, i.e., FLFFN(X), is obtained. i ′ n =Concat[X′ f ,X s ′]+X i ′ n .

[0044] The beneficial effects of this invention are:

[0045] 1. This invention proposes a novel conditional diffusion framework specifically designed for accurate pixel-level image restoration. It demonstrates state-of-the-art (SOTA) performance in multiple image restoration tasks, surpassing previous methods in the detail and structural integrity of the restored images.

[0046] 2. By introducing an enhanced training strategy, this invention improves the detail recovery fidelity of the diffusion model;

[0047] 3. This invention proposes a multi-scale frequency spatial interaction module (MFSIM) for effective dual-domain local feature fusion. This invention also integrates wavelet transform and FFT into the same framework, which retains the global frequency content of FFT and has the multi-scale analysis capability of wavelet transform.

[0048] 4. The dual-domain interactive Transformer of this invention incorporates dual-domain self-attention (DDSA) and frequency-learnable feedforward network (FLFFN) to capture long-distance dependencies between the two domains, effectively enhancing model performance.

[0049] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description

[0050] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will now be described in further detail with reference to the accompanying drawings, wherein:

[0051] Figure 1 This is a schematic diagram of the overall framework structure of Reffusion;

[0052] Figure 2 This is a schematic diagram of a multi-scale frequency space interaction module;

[0053] Figure 3 (a) is a schematic diagram of a two-domain interactive Transformer block;

[0054] Figure 3 (b) is a schematic diagram of a dual-domain self-attention unit;

[0055] Figure 3 (c) is a schematic diagram of a frequency-learnable feedforward network;

[0056] Figure 4 These are the results of a qualitative experiment on low-light enhancement.

[0057] Figure 5 These are the results of a qualitative experiment on rain streak removal;

[0058] Figure 6 These are the results of a qualitative experiment on raindrop removal;

[0059] Figure 7 These are the results of a qualitative experiment on image desnow removal;

[0060] Figure 8 This is a visualization of target detection results at night;

[0061] Figure 9 This is the result of an ablation comparison experiment to enhance training;

[0062] Figure 10 This is a comparison of results with different sampling steps. Detailed Implementation

[0063] The preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be understood that the preferred embodiments are for illustrative purposes only and are not intended to limit the scope of protection of the present invention.

[0064] This invention provides an image restoration method based on an enhanced conditional diffusion model using a dual-domain interactive Transformer, the method comprising:

[0065] Step 1: Perform diffusion training, which involves forward noise addition and reverse denoising on the clean image x0, and using the degraded image during the reverse denoising process. As a condition for guidance;

[0066] Step 2: Perform reinforcement training, using only the inverse process, i.e., Gaussian noise. A noise estimation network is used for denoising to obtain a clean image. Using degraded images during denoising Use it as a condition for guidance, and set the loss function As the objective function for the enhanced training phase, the parameter θ is further optimized;

[0067] Step 3: Use the trained noise estimation network to restore the degraded image to be restored.

[0068] The diffusion model is essentially a Markov chain, consisting of a forward diffusion (i.e., forward noise addition) process and a backward denoising process. For clean images... Using the variance sequence {β1,β2,…,β} T Gaussian noise is added forward to generate a noisy image x after T time steps. T Noisy image x T Approximate Gaussian noise, H and W represent the height and width of the image, and C represents the number of image channels, typically 3.

[0069] The process of positive noise addition can be expressed by the formula:

[0070]

[0071]

[0072] Where q represents the conditional probability distribution and t represents the time step. Let I represent a Gaussian distribution, and let x represent the identity matrix. t This represents the image at time step t.

[0073] The reverse denoising process is based on Begin learning Gaussian noise reduction transformation, and gradually work out from pure Gaussian noise. Restored to a clear image

[0074] The denoising process can be expressed by the following formula:

[0075]

[0076]

[0077] in, It is the final, clear image. This represents the image sampled at time t.

[0078] Since the original diffusion model is unconditional and the details in the generated image are uncontrollable, it is unsuitable for image restoration tasks. Therefore, this invention introduces degraded images. As a condition to guide the diffusion model. (Introducing conditional inputs) Subsequently, the distribution of the final sampled images highly conforms to the condition. This enables pixel-level image restoration.

[0079] Using degraded images After being introduced as a condition, at time step t, the mean of the noise estimation network is [value], and the inverse denoising process can be re-expressed using the formula:

[0080]

[0081]

[0082] in, To optimize the parameters of the noise estimation network, this invention reparameterizes... α t =1-β t , This represents the sampled image at time step t. This represents the noise variance at time step t. This represents the mean of the noise estimation network at time step t. This represents the noise estimated by the noise estimation network at time step t.

[0083] The ultimate training objective can be expressed as:

[0084]

[0085] Experiments show that the conditional diffusion model achieves higher structural similarity (SSIM), indicating superior performance in overall image structure restoration. However, due to insufficient pixel-level detail restoration and low fidelity, its peak signal-to-noise ratio (PSNR) is low. Therefore, this invention introduces augmentation training after the diffusion model training to improve pixel and detail restoration performance. In the augmentation training phase, only the reverse process is used to sample clean images from degraded images, and the network parameters θ are further optimized by calculating the loss between the sampled images and the real images. The novel conditional diffusion framework proposed in this invention is also called the Reffusion framework. The overall framework of Reffusion is as follows: Figure 1 As shown. The training and inference processes are described in detail in Algorithms 1 and 2, respectively.

[0086] During diffusion training, a loss function is used. As the objective function, through continuous training, it becomes... Minimum, of which in, This represents calculating the mean, ||||1 represents the L1 norm, and ε t This represents the noise added during the positive noise addition at time step t. This represents the noise estimated by the noise estimation network at time step t.

[0087] In reinforcement training, only the inverse process is used, i.e., Gaussian noise. A noise estimation network is used for denoising to obtain a clean image. Using degraded images during denoising This serves as a condition for guidance, and the weighted mixed loss function is used. As the objective function for the enhanced training phase, the parameter θ is further optimized.

[0088] in,

[0089]

[0090]

[0091] y represents the degraded image. The true clean image, SSIM(x′,y′) represents the structural similarity between any two images x′ and y′, μ x and μ y Let x and y represent the means of the images, respectively. and Let σ represent the variances of images x′ and y′, respectively. xyThe covariance of images x′ and y′ is represented by C1 and C2, which are constants used to prevent instability when the denominator is close to zero. α is a hyperparameter that can be set to 0.15.

[0092]

[0093]

[0094] In some embodiments of the present invention, the noise estimation network is a dual-domain interactive transformer (DDIT) structure.

[0095] like Figure 1 The illustrated dual-domain interactive transformer structure, arranged sequentially according to the signal input direction, includes a 3×3 first convolutional layer, an MFSIM module, seven stacked DDITB modules, and a 3×3 second convolutional layer. The stacked DDITB modules and the second convolutional layer constitute a hierarchical codec. The input to the dual-domain interactive transformer structure is noise. The noise image x obtained at time step t t With conditions The connection obtained

[0096] The output of the MFSIM module serves as the input to the first DDITB module, the output of the first DDITB module serves as the input to the second DDITB module, the output of the second DDITB module serves as the input to the third DDITB module, and the output of the third DDITB module serves as the input to the fourth DDITB module. Simultaneously, the outputs of the first and sixth DDITB modules are concatenated to serve as the input to the seventh DDITB module; the outputs of the second and fifth DDITB modules are concatenated to serve as the input to the sixth DDITB module; the outputs of the third and fourth DDITB modules are concatenated to serve as the input to the fifth DDITB module; and the output of the seventh DDITB module is input to a 3×3 second convolutional layer. After a series of processing steps, the residual image is obtained. The output of the second convolutional layer (i.e., the residual image) and x... t The final noise is obtained by adding elements one by one. The first three DDITB modules are encoders, the middle one is the bottleneck layer, and the last three DDITB modules are decoders. Decoders and encoders at the same level are connected by skip connections. Specifically, the first and seventh DDITB modules are also referred to as the first-level encoder and decoder, respectively; the second and sixth DDITB modules are also referred to as the second-level encoder and decoder, respectively; the third and fifth DDITB modules are also referred to as the third-level encoder and decoder, respectively; and the fourth DDITB module is the bottleneck layer. Each level of encoder / decoder can contain any suitable number of DDITBs, such as 4, 6, or 8. For ease of illustration, only one DDITB is shown for each DDITB module in the figure, which does not constitute a limitation of the invention. In some embodiments, the number of encoders and decoders at the first level can be 4, that is, the first DDITB module contains 4 DDITBs to form an encoder, and the seventh DDITB module contains 4 DDITBs to form a decoder; the number of encoders and decoders at the second level can be 6, that is, the second DDITB module contains 6 DDITBs to form an encoder, and the sixth DDITB module contains 6 DDITBs to form a decoder; the number of encoders and decoders at the third level can be 6, that is, the third DDITB module contains 6 DDITBs to form an encoder, the fifth DDITB module contains 6 DDITBs to form a decoder; and the fourth DDITB module contains 8 DDITBs.

[0097] The MFSIM module, or Multi-Scale Frequency Spatial Interaction Module, is used to acquire fused features of the frequency and spatial domains at multiple scales. Figure 2 This is a schematic diagram of a multi-scale frequency spatial interaction module. (Example) Figure 2 As shown, the MFSIM module sampled by this invention includes a frequency domain unit and a spatial unit, thereby employing a multi-scale design in both the spatial and frequency domains to enhance local representation.

[0098] Specifically, in the frequency domain, wavelet transform (e.g., 2D-DWT or Haar wavelet) is used to obtain multi-scale frequency domain features. This decomposes and downsamples the original image into low-frequency and high-frequency subbands XLL, {XLH, XHL, XHH} using a set of low-pass and high-pass filters. XLL represents the low-frequency subband, containing approximate information about the input image. {XLH, XHL, XHH} represent the high-frequency subbands in the horizontal, vertical, and diagonal directions, respectively, containing details of image structure and texture. To fully utilize the high- and low-frequency decomposition capabilities of wavelet transform, this invention combines convolution with discrete wavelet transform to guide the network in exploring different frequency subbands.

[0099] The spatial unit consists of three branches, which use convolutional kernels of sizes 3×3, 5×5 and 7×7 to extract spatial features at multiple scales. The output of the frequency domain unit is used as the input of the spatial unit, thereby realizing the interaction and fusion of features in the frequency domain and the spatial domain.

[0100] In some embodiments of the present invention, the DDITB module (e.g.) Figure 3 (a)) Includes dual-domain self-attention units (e.g. Figure 3 (b) and frequency-domain learnable feedforward networks (such as...) Figure 3 (b)).

[0101] The DDITB module, or Dual Domain Interaction Transformer Block, is also known as a two-domain interactive Transformer block. Since noise estimation in the diffusion model is highly correlated with the time step, this invention incorporates the time step t into the DDITB, embedding it as input. Figure 3 As shown in (a), this module processes the input feature F in ∈R H×W×C Layer normalization is performed, and sinusoidal position coding is used to convert time step t into a time embedding T′∈R. 1×1×C The input features F after layer normalization in The input is added to the temporal embedding T′ and fed into the dual-domain self-attention unit. The dual-domain self-attention unit performs attention calculations on the input in the frequency domain and spatial domain respectively, and obtains the output of the dual-domain self-attention unit. After layer regularization, the output of the dual-domain self-attention unit is fed into the frequency domain learnable feedforward network together with the temporal embedding T′.

[0102] The calculation process of the DDITB module can be expressed by the following formula:

[0103] F in =DDSA(LN(F) in )+T′)+F in ,

[0104] F out =FLFFN(LN(F in ′)+T′)+F in ′,

[0105] Among them, F in LN(·) represents the input to the DDITB module, LN(·) represents layer normalization, DDSA(·) represents the operation of the dual-domain self-attention unit, FLFFN(·) represents the operation of the frequency-domain learnable feedforward network, and F in ′ represents the output of the two-domain self-attention unit, F out This indicates the output of the DDITB module.

[0106] To comprehensively explore the characteristics of the frequency and spatial domains, this invention designs a dual-domain self-attention unit, namely DualDomain Self-Attention (DDSA), such as... Figure 3 As shown in (b).

[0107] DDSA performs attention calculations on the input in both the frequency and spatial domains, specifically including:

[0108] Input features X of the two-domain self-attention unit in ( Among them, X in =LN(F in )+T′) and time embedding Three matrices—query Q, key K, and value V—are generated using 1×1 pointwise convolution and 3×3 depthwise convolution.

[0109] The calculation process of dual-domain self-attention can be expressed by the following formula:

[0110] DDSA = Concat[DDSA] f (Q,K,V),DDSA s (Q,K,V)]+X in ,

[0111] in,

[0112]

[0113]

[0114]

[0115]

[0116]

[0117] DDSA f (·) and DDSA s (·) represents the DDSA operation in the frequency domain and spatial domain, respectively. Concat[·] represents the concatenation operation. W p and W d These represent a 1×1 pointwise convolution kernel and a 3×3 depthwise convolution kernel, respectively, with W superscripted as Q, K, and V. p and W d In the equations, β represents the weights used to calculate the query Q, key K, and value V, respectively, and β represents a learnable scaling parameter used to control the magnitude of the dot product of K and Q. Given that convolution in the spatial domain is equivalent to point-to-point multiplication in the frequency domain, this invention uses Hadamard multiplication (⊙) for the operations between Q, K, and V in the frequency domain. This represents the Fast Fourier Transform (FFT). This represents the Inverse Fast Fourier Transform (IFFT). This represents the conjugate transpose of the Fast Fourier Transform (FFT), which converts query Q and key K into frequency representations using the FFT.

[0118] Feed-forward networks (FFNs) are a key component in converters. However, not all low-frequency and high-frequency information is helpful for noise estimation, and simple FFNs cannot effectively filter out this information. To address this issue, this invention proposes a Frequency Learnable Feed-Forward Network (FLFFN), such as... Figure 3 As shown in (c), by introducing a learnable weight, FLFFN can adaptively determine which effective frequency features should be retained. Similar to DDSA, the computation of FLFFN is also performed in both the frequency and spatial domains. Specifically, the input is split along the channels, with half of it fed into a linear layer and transformed to the frequency domain via FFT. The resulting frequency features are then multiplied element-wise with a learnable weight to adapt to the frequency feature space and data distribution to select effective frequency features. They are then transformed back to the spatial domain via IFFT. After GELU activation and a linear layer, these recovered features are concatenated with the spatial features to form the final output.

[0119] Specifically, the frequency domain learnable feedforward network (FLFFN) performs the following operations:

[0120] Enter X i ′ n After layer normalization and temporal embedding, it is divided into X. f and X s , that is, X f ,X s =Split(LN(X) i ′ n )+T′), Split(·) represents the splitting operation;

[0121] X f The input is fed into a linear layer and transformed to the frequency domain using FFT. The frequency domain result is then multiplied element-wise with the learnable weights W, followed by IFFT to transform it back to the spatial domain. Finally, it is activated by GELU and processed through a linear layer to generate X′. f ,Right now This represents the Fast Fourier Transform (FFT). Represents Inverse Fast Fourier Transform (IFFT), Conv 1×1 This represents a 1×1 convolution in a linear layer;

[0122] X s X is generated after passing through a linear layer, GELU activation, and another linear layer in sequence. s ′, i.e., X s ′=Conv 1×1 (GELU(Conv 1×1 (X s )));

[0123] X′ f and X s Connect the '' and add the input X. i ′ n The output of the frequency-domain learnable feedforward network, i.e., FLFFN(X), is obtained. i ′ n =Concat[X′ f ,X s ′]+X i ′ n .

[0124] In some embodiments of the invention, a series of experiments were also conducted using PyTorch on a system equipped with an NVIDIA GeForce RTX 3090 GPU, employing the AdamW optimizer. During the diffusion training phase, the initial learning rate was set to 2 × 10⁻⁶. -4 The minimum learning rate is set to 1×10. -6 The batch size used is 16. During the augmentation training phase, the initial learning rate is set to 1×10⁻⁶. -5 The minimum learning rate is set to 1×10. -7 The batch size used was 4. The entire training process used cosine annealing scheduling, totaling 370,000 iterations. 64×64 patches were randomly cropped from the images for training. The number of encoders and decoders at each level was set to {4,6,6,8}, and the number of heads was set to {1,2,4,8}. The total time step was 1000, and the forward process used linear variance scheduling [0.0001,0.02].

[0125] The methods of this invention were evaluated on low-light image enhancement, image deraining, and snow removal tasks. For low-light image enhancement, each method was trained and tested on the LOL, LOLv2-real, and LOLv2-synthetic datasets. For image deraining, rain streaks were removed using the Rain200H and Rain200L datasets, and raindrops were removed using RainDrop. For image desnow removal, the Snow100K-S and Snow100K-L datasets were used for evaluation. The performance of all methods on the above paired datasets was quantitatively evaluated from the perspectives of pixels and structure using PSNR and SSIM. Higher PSNR and SSIM values ​​indicate higher quality and fidelity of the restored image. Furthermore, weights pre-trained on LOL were used for comparison on no-reference real datasets, including the DICM, NPE, MEF, and LIME datasets. The Natural Image Quality Evaluator (NIQE) was used to evaluate the perceptual quality of the no-reference results. NIQE reflects the naturalness of the image, with lower scores indicating higher quality. In addition, to evaluate the performance of the method of the present invention in advanced vision tasks, object detection in low-light environments was performed on the ExDark dataset.

[0126] Table 1 presents the quantitative results on low-light image enhancement datasets. The method of this invention achieves state-of-the-art performance on all three datasets. In Table 1, the arrows below each dataset in the first row indicate that larger values ​​indicate better performance of the corresponding method. Compared to other methods, the PSNR and SSIM scores of this invention are at least 1.75 dB and 0.034 higher on LOL, 0.18 dB and 0.047 higher on LOLv2-real and LOLv2-synthetic, and 0.75 dB and 0.036 higher on LOLv2-synthetic.

[0127] Table 2 presents a quantitative comparison on four unreferenced real-world datasets. In Table 2, the arrows below each dataset in the first row indicate that a smaller NIQE value indicates better performance of the corresponding method. The NIQE scores were calculated using the default NIQE function in Matlab R2020b. Compared to the second-best method, the method of this invention reduces the NIQE by 0.45. The results on the unreferenced datasets demonstrate that the method of this invention has better generalization ability for unseen scenarios.

[0128] Table 1. Quantitative Results of Low-Light Image Enhancement

[0129]

[0130]

[0131] Table 2. Quantitative results without reference dataset

[0132]

[0133] Figure 4 These are the results of the qualitative experiment on low-light enhancement, where 4(a) corresponds to the LOL dataset; 4(b) corresponds to the LOLv2-real dataset; and 4(c) corresponds to the LOLv2-synthetic dataset.

[0134] Figure 4 Qualitative results show that URetinex-Net and SCI exhibit inaccurate exposure and color distortion, while SNR-Net and Retinexformer suffer from image blurring and insufficient detail. In contrast, the method of this invention (labeled "Ours") performs excellently in exposure control, detail recovery, and color restoration, closely resembling real-world conditions. Based on quantitative and qualitative comparisons, it is clear that the method of this invention provides superior restoration results in both real-world and synthetic scenes, both at the structural and pixel levels.

[0135] Table 3 presents the quantitative experimental results for rain stripe removal. It should be noted that the arrows below each dataset in the first row of Table 3 indicate that larger values ​​indicate better performance of the corresponding method (arrows in other tables have similar meanings: upward arrows indicate larger values ​​and better performance, while downward arrows indicate smaller values ​​and better performance). As shown in Table 3, the method of this invention achieves state-of-the-art performance on the rain stripe dataset. Compared to the recent best method, DRSformer, the method of this invention improves PSNR by 0.17 dB on Rain200L and by 0.001 on SSIM; and improves PSNR by 0.37 dB on Rain200H and by 0.0065 on SSIM.

[0136] Table 3. Quantitative experimental results of rain streaks removal.

[0137]

[0138] Figure 5 The visual contrast of removing rain streaks is shown, among which, Figure 5 (a) corresponds to the Rain200H dataset; Figure 5 (b) corresponds to the Rain200L dataset. From Figure 5As can be seen, in the images from DualGCN and Restormer, the pixels at the boundaries of the zebra stripes are blurred and contain artifacts. Thanks to the dual-domain noise estimation network, Reffusion performs exceptionally well in generating superior detail, as indicated by the results labeled "Ours". This advantage is particularly evident in its outstanding rain streak removal, characterized by the sharpest edges, the highest level of detail restoration, and exceptional clarity, without any residual rain streaks.

[0139] Table 4 presents the quantitative results on the raindrop dataset. As shown in Table 4, the method of this invention is the most effective in removing raindrops. Compared to the second-best method, PSNR is increased by 0.25 dB and SSIM is increased by 0.0033.

[0140] Table 4. Quantitative experimental results of raindrop removal.

[0141]

[0142]

[0143] Figure 6 Visual results are shown on the RainDrop dataset. The method of this invention (i.e., labeled "Ours") excels in thoroughly removing raindrops, especially in areas heavily affected by raindrops, surpassing other methods.

[0144] Table 5 presents the quantitative experimental results of image desnowing. The results shown in Table 5 verify the state-of-the-art (SOTA) performance of the method of this invention in the field of image desnowing. On Snow100K-S, the PSNR and SSIM of the method of this invention are significantly better than other methods, exceeding them by 0.45 dB and 0.0045, respectively; on Snow100K-L, they are 0.48 dB and 0.0151 higher, respectively.

[0145] Table 5. Results of the quantitative snow removal experiment in images.

[0146]

[0147] Figure 7 Qualitative results are shown on Snow100K-L. Compared to SnowDiff64, the method of this invention (i.e., marked "Ours") restores images that are closer to reality, with sharper edges, better color restoration, and more refined detail restoration.

[0148] To evaluate the effectiveness of the method of this invention in high-level vision, low-light object detection was performed on the ExDark dataset. This dataset contains images of 12 categories, each characterized by underexposure, with corresponding annotations. In this embodiment, it is divided into training, validation, and test sets in a ratio of 8:1:1, and YOLO-v5 is used as the detection network. Different methods are pre-trained on the LOL dataset as a preprocessing module of YOLO-v5. The results on ExDark are shown in Table 6.

[0149] Table 6 Detection results of low-light targets

[0150]

[0151] As can be seen from Table 6, the method of the present invention achieved the highest mean accuracy (mAP) across all categories, with an improvement of at least 0.025. The method of the present invention outperformed other methods in 10 of the 12 categories, performing only slightly lower in the chair and table categories.

[0152] Figure 8 Qualitative results of nighttime object detection visualization on ExDark are presented. From Figure 8 As can be seen, the SCI results are overexposed, while SNR-Net is inferior to the method of this invention in terms of detail and clarity (i.e., marked "Ours"). It is worth noting that using Reffusion enables the detection network to accurately detect all objects, providing precise bounding box predictions. Results on the ExDark dataset demonstrate the effectiveness and generalization ability of the method of this invention in high-level vision.

[0153] To verify the effectiveness of DDSA, FLFFN, and MFSIM, some embodiments of this invention used Restormer as baseline V1 and established a control group for ablation experiments. It should be noted that all ablation experiments were performed on the Rain200H dataset and maintained the same experimental settings as described above.

[0154] Table 7 presents the effectiveness validation results of MFSIM, DDSA, and FLFFN. The results in Table 7 show that the recovery performance was significantly improved after applying DDSA, FLFFN, and MFSIM. Compared to the baseline, the method of this invention improved PSNR by 1.06 dB and SSIM by 0.0076 dB, thus validating the effectiveness of DDSA, FLFFN, and MFSIM.

[0155] Table 7. Validation results of MFSIM, DDSA, and FLFFN

[0156] V1 × × × 31.3 0.9295 V2 √ × × 31.57 0.9297 V3 × √ × 32.13 0.9353 V4 × × √ 32.07 0.9322 V5 × √ √ 32.36 0.9371 V6 √ √ √ 32.55 0.9395

[0157] To verify the effectiveness of the augmentation training, comparative experiments were also conducted in some embodiments of the present invention. Figure 9 The visualization comparison results of augmentation training are presented. After applying augmentation training, the PSNR value increased from 31.89 dB to 32.55 dB, and the SSIM increased from 0.9326 to 0.9395. This indicates that augmentation training can effectively improve the fidelity and recovery performance of Reffusion.

[0158] To explore the impact of sampling step size S on recovery performance, some embodiments of the present invention use SNR, SSIM and sampling time as evaluation indicators to conduct quantitative comparison experiments of different values ​​of S. Figure 10 This is a comparison of different sampling steps; the size of the bubbles in the figure represents the required sampling time. Figure 10 The results shown indicate that optimal recovery performance occurs at S=3, while the required sampling time is relatively short. Therefore, S=3 can be selected as the default setting to achieve excellent recovery performance and relatively fast sampling speed.

[0159] Therefore, this invention first proposes an enhanced conditional diffusion model framework for image restoration called Reffusion by improving the noise estimation accuracy in the diffusion model. This framework incorporates a two-domain interactive transformer. Then, a conditional diffusion model with an enhanced training strategy is introduced to achieve fine-grained image restoration. Through the interaction between two-domain features, the method of this invention ensures accurate noise estimation, thereby improving the performance of the diffusion model in image restoration. Quantitative and qualitative experiments show that Reffusion exhibits state-of-the-art performance in three image restoration tasks. Furthermore, extending the evaluation to low-light object detection verifies the effectiveness of the method in advanced vision. The method of this invention provides a promising direction for improving the performance and applicability of multimedia in diverse and complex scenes.

[0160] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. An image restoration method based on a dual-domain interactive Transformer-based enhanced conditional diffusion model, characterized in that, include: Step 1: Perform diffusion training, which involves forward noise addition and reverse denoising on the clean image x0, and using the degraded image during the reverse denoising process. As a conditional guide, a noise estimation network with parameter θ is used in the reverse denoising process. At time step t, the mean of the noise estimation network is... in, This represents the sampled image during the reverse denoising process at time step t. During diffusion training, a loss function is used. As the objective function, through continuous training, it becomes... Minimum, of which in, This represents calculating the mean, |||1 represents the L1 norm, and ε t This represents the noise added during the positive noise addition at time step t. This represents the noise estimated by the noise estimation network at time step t; Step 2: Perform reinforcement training, i.e., Gaussian noise A noise estimation network is used for denoising to obtain a clean image. Using degraded images during denoising Use it as a condition for guidance, and set the loss function As the objective function for the enhanced training phase, the parameter θ is further optimized. in, y represents the degraded image. The true clean image, SSIM(x′,y′) represents the structural similarity between any two images x′ and y′, μ x and μ y Let x and y represent the means of the images, respectively. and Let σ represent the variances of images x′ and y′, respectively. xy The covariance of images x′ and y′ is represented by C1 and C2, which are constants used to prevent instability when the denominator approaches zero. α is a hyperparameter. This is the clean image obtained from the final sampling; Step 3: Using the trained noise estimation network, perform image restoration on the degraded image to be restored. The noise estimation network is a dual-domain interactive transformer structure, which includes a 3×3 first convolutional layer, an MFSIM module, seven stacked DDITB modules, and a 3×3 second convolutional layer in sequence according to the signal input direction. The MFSIM module is used to obtain multi-scale frequency domain and spatial domain fusion features. The output of the first DDITB module is concatenated with the output of the sixth DDITB module as the input of the seventh DDITB module, the output of the second DDITB module is concatenated with the output of the fifth DDITB module as the input of the sixth DDITB module, and the output of the third DDITB module is concatenated with the output of the fourth DDITB module as the input of the fifth DDITB module.

2. The image restoration method based on the enhanced conditional diffusion model of a dual-domain interactive Transformer according to claim 1, characterized in that, Apply a variance sequence {β1,β2,…,β} to the clean image x0. T Gaussian noise is added forward to generate a noisy image x after T time steps. T Noisy image x T Approximate Gaussian noise, the process of positive noise addition can be expressed by the following formula: Where q represents the conditional probability distribution and t represents the time step. Let I represent a Gaussian distribution, and let x represent the identity matrix. t This represents the image at time step t.

3. The image restoration method based on a dual-domain interactive Transformer-based enhanced conditional diffusion model according to claim 1, characterized in that, The reverse denoising process is based on Start learning Gaussian denoising transformation and using degraded images. As a condition, the reverse denoising process is expressed by the following formula: in, α t =1-β t , This represents the sampled image at time step t. This represents the noise variance at time step t. This represents the mean of the noise estimation network at time step t. This represents the noise estimated by the noise estimation network at time step t.

4. The image restoration method based on the enhanced conditional diffusion model of a dual-domain interactive Transformer according to claim 1, characterized in that, The MFSIM module includes a frequency domain unit and a spatial unit. In the frequency domain unit, wavelet transform is used to obtain multi-scale frequency domain features. The spatial unit consists of three branches, which use convolutional kernels of sizes 3×3, 5×5 and 7×7 to extract spatial features at multiple scales. The output of the frequency domain unit is used as the input of the spatial unit, thereby realizing the interaction and fusion of features in the frequency domain and the spatial domain.

5. The image restoration method based on the enhanced conditional diffusion model of a dual-domain interactive Transformer according to claim 1, characterized in that, The DDITB module includes a dual-domain self-attention unit and a frequency-domain learnable feedforward network. The DDITB module processes the input feature F. in Layer normalization is performed, and sinusoidal positional encoding is used to convert time step t into temporal embedding T′. The input feature F after layer normalization is then processed. in The input is added to the temporal embedding T′ and fed into a dual-domain self-attention unit. The dual-domain self-attention unit performs attention calculations on the input in both the frequency and spatial domains, yielding its output. After layer regularization, the output of the dual-domain self-attention unit, along with the temporal embedding T′, is then input into a frequency-domain learnable feedforward network. The calculation process of the DDITB module can be expressed by the following formula: F′ in =DDSA(LN(F in )+T′)+F in , F out =FLFFN(LN(F′ in )+T′)+F′ in , Among them, F in LN(·) represents the input to the DDITB module, LN(·) represents layer normalization, DDSA(·) represents the operation of the dual-domain self-attention unit, FLFFN(·) represents the operation of the frequency-domain learnable feedforward network, and F′ represents the input to the DDITB module. in F represents the output of the two-domain self-attention unit. out This indicates the output of the DDITB module.

6. The image restoration method based on the enhanced conditional diffusion model of a dual-domain interactive Transformer according to claim 5, characterized in that, The dual-domain self-attention unit performs attention calculations on the input in both the frequency and spatial domains, specifically including: Input features X of the two-domain self-attention unit in Temporal embedding is used to generate three matrices: query Q, key K, and value V, by applying 1×1 pointwise convolution and 3×3 depthwise convolution. The calculation process of dual-domain self-attention is expressed by the following formula: DDSA=Concat[DDSA f (Q,K,V),DDSA s (Q,K,V)]+X in , in, DDSA f (·) and DDSA s (·) represents the DDSA operation in the frequency domain and spatial domain, respectively; Concat[·] represents the concatenation operation; W p and W d These represent a 1×1 pointwise convolution kernel and a 3×3 depthwise convolution kernel, respectively, with W superscripted as Q, K, and V. p and W d In the equations, represent the weights when calculating query Q, key K, and value V, respectively, and β represents the learnable scaling parameter. This represents the Fast Fourier Transform (FFT). This represents the Inverse Fast Fourier Transform (IFFT). denoted by , and ⊙ denotes the conjugate transpose of the Fast Fourier Transform.

7. The image restoration method based on a dual-domain interactive Transformer-based enhanced conditional diffusion model according to claim 5, characterized in that, Frequency-domain learnable feedforward networks perform the following operations: Input X′ in After layer normalization and temporal embedding, it is divided into X. f and X s , that is, X f ,X s =Split(LN(X′) in )+T′), Split(·) represents the splitting operation; X f The input is fed into a linear layer and transformed to the frequency domain using FFT. The frequency domain result is then multiplied element-wise with the learnable weights W, followed by IFFT to transform it back to the spatial domain. Finally, it is activated by GELU and processed through a linear layer to generate X′. f ,Right now This represents the Fast Fourier Transform (FFT). Represents Inverse Fast Fourier Transform (IFFT), Conv 1×1 ⊙ represents a 1×1 convolution of a linear layer, and ⊙ represents the Hadamard product. X s X′ is generated after passing through a linear layer, GELU activation, and another linear layer in sequence. s , i.e., X′ s =Conv 1×1 (GELU(Conv 1×1 (X s ))); X′ f and X′ s Connect the components, and add the input X′. in The output of the frequency-domain learnable feedforward network, i.e., FLFFN(X′), is obtained. in =Concat[X′ f ,X′ s ]+X′ in .

Images