A universal image restoration method and system based on dual-space joint diffusion

Abstract

The present application provides a general image restoration method and system for dual-space joint diffusion, which is used to solve the problem that existing image restoration methods are difficult to achieve comprehensive image restoration under different degradation modes. The method steps are: constructing a dual-space diffusion model DSDM with a degraded image as input and a true value image as output; training the DSDM model using an image dataset; restoring the collected degraded image through the trained model; the DSDM model includes a MU-Net network and a fusion refinement network FRN, and the MU-Net network includes a first and a second U-net convolutional neural network that perform diffusion processes in pixel space and latent space respectively. The fusion refinement network FRN is used to fuse the outputs of the two U-net convolutional neural networks and output a restored image. The present application constructs a MU-Net network to diffuse the degraded image in the pixel space and latent space respectively. The two sets of diffusion results are fused through the fusion refinement network FRN, and the restored image output has high quality.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a universal image restoration method and system using dual-space joint diffusion. Background Art

[0002] Image quality often degrades during acquisition, storage, transmission, and rendering, making image restoration technology particularly important. The goal of image restoration technology is to restore a high-quality, clear image from a degraded image as accurately as possible. Traditional image restoration methods are typically designed for specific degraded image restoration tasks, such as dehazing, rain removal, deblurring, and enhancing low-light images. However, in practical applications, image degradation forms are often complex and varied, such as haze, rain, snow, and low light, and are difficult to predict. This results in poor performance of single-task methods in practical application scenarios, with problems such as high storage costs, lack of flexibility, and even failure.

[0003] Therefore, universal image restoration (UIR) methods have emerged. These methods aim to address multiple degradation types using a unified model. The core challenge of UIR methods lies in how to simultaneously optimize the model to learn multiple degradation tasks while avoiding conflicts between them. Early work tended to design separate sub-networks for each task, but this limited the flexibility and scalability of the methods. To this end, some studies have attempted to adapt the classic image generation paradigm for universal image restoration. Diffusion models, due to their excellent generative capabilities and robustness, have gradually become the backbone of this field. Generally speaking, diffusion-based UIR methods can be divided into two categories: the first category is cue-based diffusion methods. These methods encode the degraded image as cue information to guide the inverse diffusion process to achieve a specific restoration task. However, these methods explicitly model different degradation tasks separately, ignoring the inherent connections between them. The second category is cue-free unified diffusion methods. These methods directly construct a unified forward and backward diffusion process for each degraded image in pixel space or latent space, without the need for cue information. Although this method can stimulate the model to actively explore the connections between different tasks, it requires the diffusion model to autonomously adapt to various degradation patterns without any discriminative clues. The information in a single space often cannot support this challenging purpose, resulting in poor recovery ability. Summary of the Invention

[0004] The purpose of the present invention is to provide a universal image restoration method and system of dual-space joint diffusion, which is used to solve the technical problem that existing image restoration methods are difficult to achieve full image restoration under different degradation modes.

[0005] A universal image restoration method based on dual-space joint diffusion, the specific steps are as follows:

[0006] S1: Construct a dual-space diffusion model DSDM with degraded image as input and true image as output;

[0007] S2: Use the image dataset to train the dual-space diffusion model DSDM;

[0008] S3: Restore the collected degraded image using the trained model;

[0009] The dual-space diffusion model DSDM includes a MU-Net network and a fusion refinement network FRN; the MU-Net network includes a first and a second U-net convolutional neural network that perform diffusion processes in pixel space and latent space, respectively, and the fusion refinement network FRN is used to fuse the outputs of the first and second U-net convolutional neural networks and output a restored image.

[0010] Optionally, both the pixel space diffusion and the latent space diffusion of the degraded image include a forward diffusion process and a backward diffusion process, wherein the backward diffusion process takes the output of the forward diffusion process as input.

[0011] Optionally, the first and second U-net convolutional neural networks respectively include a latent feature-based pixel feature enhancement module L2P and a pixel feature-based latent feature enhancement module P2L;

[0012] The latent feature based pixel feature enhancement module L2P is used to select information from the latent space back diffusion process to enhance the pixel space back diffusion process, and the pixel feature based latent feature enhancement module P2L is used to select information from the pixel space back diffusion process to enhance the latent space back diffusion process.

[0013] Optionally, the specific method of the latent feature-based pixel feature enhancement module L2P selecting information from the latent space back diffusion process to enhance the pixel space back diffusion process is:

[0014] Pixel spatial features of the k-th layer decoder in the first and second U-net convolutional neural networks and latent space features Normalization is performed and the normalized pixel space features are Perform downsampling to obtain downsampled features ;

[0015] Normalize the latent space features and downsampled features Perform channel splicing to form preliminary fusion features, and then extract the preliminary fusion features through two layers of convolutional layers and corresponding activation functions to generate filter masks ;

[0016] Normalize the latent space features With filter mask Perform channel splicing and upsampling to obtain spliced ​​upsampled features, and then use Swin Transformer blocks and convolution layers to combine spliced ​​upsampled features and normalized pixel space features. Extract and compare the extracted features with the normalized pixel space features Fusion to obtain enhanced pixel space features .

[0017] Optionally, the specific method of the pixel feature-based latent feature enhancement module P2L selecting information from the pixel space reverse diffusion process to enhance the latent space reverse diffusion process is:

[0018] Pixel spatial features of the k-th layer decoder in the first and second U-net convolutional neural networks and latent space features Normalize and normalize the latent space features Perform upsampling to obtain upsampled features ;

[0019] Normalize pixel space features and upsampled features Perform channel splicing to form preliminary fusion features, then extract the preliminary fusion features through the convolution layer and the corresponding activation function to generate the filter mask ;

[0020] Normalize pixel space features With filter mask Perform channel splicing and downsampling to obtain spliced ​​downsampled features, and then use the self-attention mechanism block and convolution layer to combine the spliced ​​downsampled features and normalized latent space features. Extract and compare the extracted features with the normalized latent space features Fusion to obtain enhanced pixel space features .

[0021] Optionally, the specific method of the fusion refinement network FRN fusing the outputs of the first and second U-net convolutional neural networks and outputting the restored image is:

[0022] The outputs of the first and second U-net convolutional neural networks and Perform channel splicing and extract hierarchical features through convolutional layers and channel splitting modules; perform convolution or dilated convolution operations on the hierarchical features, and perform residual operations on the operated features to obtain multi-scale hierarchical features;

[0023] The multi-scale hierarchical features are channel-spliced, and then the channel-spliced ​​multi-scale hierarchical features are fused through the convolution layer, sliding window attention block, residual block and convolution layer to generate the final image.

[0024] Optionally, a specific method for training the dual-space diffusion model DSDM using an image dataset is as follows:

[0025] S2.1: Construct the loss functions of pixel space diffusion and latent space diffusion respectively, and use the sum of the loss functions of pixel space diffusion and latent space diffusion as the loss function of the MU-Net network;

[0026] S2.2: Train the MU-Net network using the image dataset as the training set data to obtain the trained MU-Net network;

[0027] S2.3: Freeze the trained MU-Net network parameters and perform T-step back diffusion to obtain the recovery results in two different diffusion spaces;

[0028] S2.4: Construct a loss function for the fusion and refinement network FRN, train the fusion and refinement network FRN with the recovery results of two different diffusion spaces as input, and obtain the trained fusion and refinement network FRN.

[0029] Optionally, the collected degraded image is restored using the trained model. The specific steps are as follows:

[0030] S3.1: Input the collected degraded image into the trained dual-space diffusion model DSDM and perform forward diffusion in pixel space and latent space respectively;

[0031] S3.2: Input the forward diffusion outputs in pixel space and latent space into the first and second U-net convolutional neural networks respectively to perform backward diffusion;

[0032] S3.3: The outputs of the first and second U-net convolutional neural networks are used as inputs of the fusion refinement network FRN, the outputs of the first and second U-net convolutional neural networks are fused through the fusion refinement network FRN, and a restored image is output.

[0033] A dual-space joint diffusion universal image restoration system, used to implement the above-mentioned dual-space joint diffusion universal image restoration method, comprising:

[0034] Model building module: used to build a dual-space diffusion model DSDM with degraded image as input and restored image as output;

[0035] Model training module: used to train the dual-space diffusion model DSDM using image datasets;

[0036] Image restoration module: used to restore the collected degraded images through the trained model;

[0037] The dual-space diffusion model DSDM includes a MU-Net network and a fusion refinement network FRN; the MU-Net network includes a first and a second U-net convolutional neural network that perform diffusion processes in pixel space and latent space, respectively, and the fusion refinement network FRN is used to fuse the outputs of the first and second U-net convolutional neural networks and output a restored image.

[0038] Due to the adoption of the above technical solution, the present invention has the following advantages:

[0039] 1. This application constructs an MU-Net network to diffuse the degraded image in the pixel space and the latent space respectively. The two sets of diffusion results are fused through the fusion refinement network FRN to output a high-quality restored image;

[0040] 2. This application integrates the diffusion processes in pixel space and latent space, and uses their complementarity to uniformly handle multiple degradation tasks, thereby reducing learning difficulty and improving generalization ability;

[0041] 3. The fusion refinement network FRN of this application fuses the restoration results from the pixel space and the latent space at multiple scales, and the generated restored image has higher quality.

[0042] Other advantages, objects, and features of the present invention will be described in part in the following description and, in part, will be apparent to those skilled in the art upon examination of the following description or may be learned from practice of the present invention. The objects and other advantages of the present invention may be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The accompanying drawings of the present invention are described below.

[0044] Figure 1 Schematic diagram of the structure of the dual-space diffusion model DSDM of the present invention.

[0045] Figure 2 Schematic diagram of the structure of the pixel feature enhancement module L2P based on latent features of the present invention.

[0046] Figure 3 Schematic diagram of the structure of the potential feature enhancement module P2L based on pixel features of the present invention.

[0047] Figure 4 This is a structural diagram of the fusion refinement network FRN of the present invention.

[0048] Figure 5This is the visualization result of the filter mask in the last layer of the L2P and P2L modules of the present invention.

[0049] Figure 6 Figure 1 is a training process diagram of the MU-Net network and the fused refinement network FRN of the present invention, where a is a training process diagram of the MU-Net network and b is a training process diagram of the fused refinement network FRN.

[0050] Figure 7 The figure is a visual comparison of the present invention and the existing methods in five tasks.

[0051] Figure 8 These are visual comparison diagrams of the present invention and general image restoration methods in four degradation tasks, where a is a visual comparison diagram of real scene rain removal in the dataset LHP, b is a visual comparison diagram of real scene snow removal in the dataset RealSnow, c is a visual comparison diagram of real scene deblurring in the dataset RealBlur-J, and d is a visual comparison diagram of defocus blur removal in the dataset DPDD.

[0052] Figure 9 These are visual comparison diagrams of the present invention and general image restoration methods in three composite degradation tasks, where a is a visual comparison diagram of UDC image restoration on the POLED dataset, b is a visual comparison diagram of UDC image restoration on the TOLED dataset, and c is a visual comparison diagram of hybrid deraining restoration on the Outdoor-Rain dataset.

[0053] Figure 10 This is the ablation diagram of the diffusion step length of the MU-Net network of the present invention on the Rain100H dataset. DETAILED DESCRIPTION

[0054] The present invention will be further described below with reference to the accompanying drawings and examples. Example 1:

[0055] like Figure 1 A general image restoration method of dual-space joint diffusion is shown, and the specific steps are as follows:

[0056] S1: Construct a dual-space diffusion model DSDM with degraded image as input and true image as output;

[0057] like Figure 1 As shown, the dual-space diffusion model DSDM includes a MU-Net network and a fusion refinement network FRN; the MU-Net network includes a first U-net convolutional neural network that performs diffusion processes in pixel space and latent space respectively. and the second U-net convolutional neural network , the fusion refinement network FRN is used to fuse the outputs of the first and second U-net convolutional neural networks and output a restored image.

[0058] In this embodiment, both pixel-space diffusion and latent-space diffusion of a degraded image include a forward diffusion process and a backward diffusion process. The backward diffusion process uses the output of the forward diffusion process as input. The forward diffusion process gradually adds Gaussian noise to the degraded image to obtain a pure noise image. The backward diffusion process uses the pure noise image as input and gradually reverses the diffusion process to restore the image.

[0059] In pixel space, the forward diffusion process expression is:

[0060] ,

[0061] Where, is the image in pixel space at time t, is the residual image, , For degraded images, is the true value image, and are the weights of the residual image and Gaussian noise, , ,when When forward diffusion is completed and output , T is the total number of diffusion steps.

[0062] In pixel space, the back diffusion process is expressed as:

[0063] ,

[0064] Where, is the true value image predicted at time t−1, and is the output of the pixel space back diffusion at time t and t-1, and the back diffusion starts from Start, final output The restored result is obtained by pixel-space diffusion.

[0065] In the latent space diffusion process, the pre-trained VQGAN encoder is used With decoder Implement pixel-latent space mapping:

[0066] ,

[0067] The above process will and Bring subtle high-frequency differences, remember 、 、 , then the forward diffusion process of the latent space is:

[0068] ,

[0069] Where, is Gaussian noise.

[0070] The back diffusion process in the latent space is:

[0071] ,

[0072] Among them hour, ; The final restored image is passed Decoded, as the output of the latent space diffusion. In this embodiment, although RDDM recommends setting different However, it is unreasonable to set specific hyperparameters for each type of degradation task in the general image restoration task. Therefore, in the MU-Net network of this application, we empirically set Set to 1.

[0073] In this embodiment, if Figure 4 As shown, the fusion refinement network FRN fuses the outputs of the first and second U-net convolutional neural networks and outputs the restored image in the following specific method:

[0074] The outputs of the first and second U-net convolutional neural networks and Perform channel splicing and extract hierarchical features through convolutional layers and channel splitting modules:

[0075] ,

[0076] Where Cat is channel splicing, is the convolution operation, Channel splitting operation, F1, F2 and F3 are three hierarchical features;

[0077] Perform convolution and residual operations on the hierarchical feature F1, and perform dilated convolution and residual operations on F2 and F3 to obtain multi-scale hierarchical features. 、 and ;

[0078] ,

[0079] Where, represents a dilated convolution operation with a dilation rate of k, is the residual operation.

[0080] Multi-scale hierarchical features 、 and Perform channel splicing, and then fuse the multi-scale hierarchical features of channel splicing through convolutional layers, sliding window attention blocks, residual blocks, and convolutional layers to generate the final image:

[0081] ,

[0082] Where, It is a sliding window attention operation; the fusion refinement network FRN of this application fuses the restoration results from the pixel space and the latent space at multiple scales, and the generated restored image has higher quality.

[0083] In this embodiment, if Figure 2 and Figure 3 As shown, the first and second U-net convolutional neural networks respectively include a latent feature-based pixel feature enhancement module L2P and a pixel feature-based latent feature enhancement module P2L; the latent feature-based pixel feature enhancement module L2P is used to select information from the latent space back diffusion process to enhance the pixel space back diffusion process, and the pixel feature-based latent feature enhancement module P2L is used to select information from the pixel space reverse direction diffusion process to enhance the latent space back diffusion process.

[0084] Specifically, the specific method of the pixel feature enhancement module L2P based on the latent feature to select information from the latent space back diffusion process to enhance the pixel space back diffusion process is as follows:

[0085] Pixel spatial features of the k-th layer decoder in the first and second U-net convolutional neural networks and latent space features Normalization is performed and the normalized pixel space features are Perform downsampling to obtain downsampled features ;

[0086] ,

[0087] Where, is the normalization function;

[0088] Normalize the latent space features and downsampled features Perform channel splicing to form preliminary fusion features, and then extract the preliminary fusion features through two layers of convolutional layers and corresponding activation functions to generate filter masks ;

[0089] ,

[0090] Where, and are the activation functions of the two convolutional layers respectively, is downsampling;

[0091] Normalize the latent space features With filter mask Perform channel splicing and upsampling to obtain spliced ​​upsampled features, and then use Swin Transformer blocks and convolution layers to combine spliced ​​upsampled features and normalized pixel space features. Extract and compare the extracted features with the normalized pixel space features Fusion to obtain enhanced pixel space features :

[0092] ,

[0093] Where, For upsampling, Refers to the Swin Transformer block, which facilitates optimization Fine-grained spatial relationships between pixels.

[0094] Specifically, the method for the pixel feature-based latent feature enhancement module P2L to select information from the pixel space reverse diffusion process to enhance the latent space reverse diffusion process is as follows:

[0095] Pixel spatial features of the k-th layer decoder in the first and second U-net convolutional neural networks and latent space features Normalize and normalize the latent space features Perform upsampling to obtain upsampled features ;

[0096] Normalize pixel space features and upsampled features Perform channel splicing to form preliminary fusion features, then extract the preliminary fusion features through the convolution layer and the corresponding activation function to generate the filter mask :

[0097] ,

[0098] Normalize pixel space features With filter mask Perform channel splicing and downsampling to obtain spliced ​​downsampled features, and then use the self-attention mechanism block and convolution layer to combine the spliced ​​downsampled features and normalized latent space features. Extract and compare the extracted features with the normalized latent space features Fusion to obtain enhanced pixel space features :

[0099] ,

[0100] Where, represents transposed attention, directly modeling self-attention across channel dimensions, thereby enhancing In the above feature interaction process, The appropriate global information is filtered from the latent space to the pixel space in a gated manner to promote the global consistency of the reconstructed features. Edges and fine texture details are also selected from the pixel space to the latent space to enrich the reconstructed features. Figure 5 The filter mask shown ( and ), It can effectively guide the semantic consistency information in the latent space to the pixel space, The details and texture information in the pixel space are transferred to the latent space. This phenomenon shows that in the diffusion process, the filter mask effectively plays a gating role in information selection and complementary regulation.

[0101] S2: If Figure 6 As shown in Figure 2, the dual-space diffusion model DSDM is trained using the image dataset; the specific steps are:

[0102] S2.1: Construct the loss functions of pixel space diffusion and latent space diffusion respectively, and use the sum of the loss functions of pixel space diffusion and latent space diffusion as the loss function of the MU-Net network;

[0103] ,

[0104] ,

[0105] Where, is the loss function of pixel space diffusion, is the loss function for latent space diffusion, is the loss function of the MU-Net network, , T is the total number of diffusion steps, is the L1 norm (sum of absolute value differences), is the mathematical expectation.

[0106] S2.2: Train the MU-Net network using the image dataset as the training set data to obtain the trained MU-Net network;

[0107] In this embodiment, the image dataset is a historical dataset that includes degraded images and true image data.

[0108] S2.3: Freeze the trained MU-Net network parameters and perform T-step back diffusion to obtain the recovery results in two different diffusion spaces;

[0109] S2.4: Construct the loss function of the fusion and refinement network FRN, and train the fusion and refinement network FRN with the recovery results of two different diffusion spaces as input to obtain the trained fusion and refinement network FRN:

[0110] ,

[0111] Where, is the loss function of the fusion refinement network FRN, and denote L1 reconstruction loss and perceptual loss respectively.

[0112] S3: Restore the collected degraded image using the trained model; the specific steps are:

[0113] S3.1: Input the collected degraded image into the trained dual-space diffusion model DSDM and perform forward diffusion in pixel space and latent space respectively;

[0114] S3.2: Input the forward diffusion outputs in pixel space and latent space into the first and second U-net convolutional neural networks respectively to perform backward diffusion;

[0115] S3.3: The outputs of the first and second U-net convolutional neural networks are used as inputs of the fusion refinement network FRN, the outputs of the first and second U-net convolutional neural networks are fused through the fusion refinement network FRN, and a restored image is output.

[0116] S4: Experiment and Verification:

[0117] S4.1: Dataset and experimental parameter settings:

[0118] General image restoration setting: The datasets of five typical image degradation tasks are combined into a unified dataset containing multiple degradation types for training and evaluation, including: Rain removal task: Rain100H dataset, 1800 training sets and 100 test sets; low-light enhancement task: LOL-v1 dataset, 485 training sets and 15 test sets; snow removal task: Snow100K dataset, 1872 training sets and 601 test sets; dehazing task: RESIDE dataset, 6000 training sets and 1000 test sets; motion deblurring task: GoPro dataset, 2103 training sets and 1111 test sets.

[0119] Zero-shot Application Setup: The trained model in this application is applied to the following real-world degradation or compound degradation tasks without additional training: rain removal on the LHP dataset; snow removal on the RealSnow dataset; motion deblurring on the RealBlur-J dataset; focus deblurring on the DPDD dataset; under-display camera image restoration on POLED and TOLED; and mixed rain removal on the Outdoor-Rain dataset. A standard segmentation process was used for all of these datasets. The specific segmentation for each dataset is shown in Table 1.

[0120] Table 1 Dataset division

[0121] ,

[0122] Comparison Method Setup: This application's image restoration method is compared with various task-specific and general-purpose image restoration methods. Task-specific methods include HDCW-Net, Restormer, WeatherDiff, LLFormer, IR-SDE, MSDT, DEA-Net, GSAD, and MambaIR; and general-purpose methods include AirNet, PromptIR, U-WADN, DA-CLIP, DiffUIR, and NDR-Restore. For all comparison models, official pre-trained weights were used whenever possible. If unavailable, pre-training was performed on the corresponding datasets based on their code and hyperparameters.

[0123] Evaluation index setting: Peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) are used as indicators of distortion performance, and perceptual loss (LPIPS) and FID are used as indicators of perceptual performance.

[0124] Experimental environment and training details: PyTorch framework was used for training on a single NVIDIA RTX 3090 GPU. In the first phase (MU-Net training), 300,000 iterations were performed with a batch size of 8. In the second phase (FRN training), 150,000 iterations were performed with a batch size of 10. The Adam optimizer was used, and the learning rate was 10 at each phase. , We randomly cropped 256×256 image patches from the original image and used horizontal / vertical flipping for data augmentation. Furthermore, we sampled the different datasets with equal probability to balance the differences in training data size. During training and inference, we set the total number of diffusion time steps in the DSDM model to T = 3.

[0125] S4.2: Performance Comparison on General Image Restoration Tasks: In a general image restoration setting, the proposed image restoration method is compared with existing image restoration methods. The results are shown in Table 2, which quantitatively compares the distortion metrics (PSNR / SSIM) and perceptual metrics (FID / LPIPS) of each method on five target tasks. Experiments show that the proposed method consistently outperforms other general image restoration methods in terms of distortion and perceptual metrics across all tasks and is competitive with most task-specific image restoration methods.

[0126] Table 2 Qualitative comparison of restoration results of various image restoration methods for different degraded images

[0127] ,

[0128] like Figure 7 As shown in Figure 3, the experimental results show that compared with other image restoration methods, the method proposed in this application maintains the fidelity of structure and texture while processing degraded content.

[0129] S4.3: Zero-shot application performance comparison: In the zero-shot application setting, the general image restoration model trained on five types of degradation tasks is directly applied to the following real degradation tasks and compound degradation tasks that have not been seen in the training process to evaluate the generalization ability of each general image restoration method.

[0130] Quantitative and qualitative comparisons are conducted on the LHP, RealSnow, RealBlur-J, and DPDD datasets for rain removal, snow removal, blur removal, and defocus blur removal in real scenes. As shown in Table 3, the proposed method shows consistent restoration performance on these real-world degradation tasks compared to existing general image restoration methods. Figure 8 As shown, Figure 8 The results show that DA-CLIP and PromptIR methods are difficult to generalize to new degradation tasks, resulting in incorrect recovery patterns. DiffUIR and NDR-Restore methods are limited in their ability to handle severely degraded content. The method proposed in this application, by combining a unified problem modeling approach with a dual-space diffusion mechanism, effectively learns and generalizes to unseen real-world degraded images, maintaining structural integrity and detail consistency.

[0131] Table 3 Performance comparison on real-world degradation tasks

[0132] ,

[0133] Further quantitative and qualitative comparative experiments were conducted on POLED, TOLED and Outdoor-Rain datasets to demonstrate the effectiveness of the general image restoration method in complex degradation scenarios. As shown in Table 4, the method proposed in this application consistently achieves superior performance compared to other methods. Figure 9 As shown, Figure 9 The results show that DA-CLIP and PromptIR methods struggle to accurately distinguish all composite degradations based on a single coding prompt, while DiffUIR and NDR-Restore methods provide insufficient recovery and even tamper with some non-degraded details. The proposed method demonstrates stronger overall recovery capabilities, accurately restoring multiple degraded regions while preserving the original appearance of non-degraded regions.

[0134] Table 4 Performance comparison on composite degradation tasks

[0135] ,

[0136] S4.4: Ablation Experiments: Ablation experiments are conducted to verify the effectiveness of key modules in DSDM, including the component design of MU-Net and FRN, the selection of diffusion prediction targets, and the setting of diffusion steps. All experiments are conducted under a general image restoration setting with the same hyperparameter configuration.

[0137] S4.4.1: Module role verification: As shown in Table 5, the impact of key components in the DSDM model on performance is compared on the Rain100H dataset. The comparison objects include: removing the L2P module in the MU-Net network, removing the P2L module, replacing the information interaction module with a simple size adjustment + splicing, not using the FRN network, and directly outputting the result of a single path as the final image. In Table 5, "replacement" refers to replacing the L2P and P2L modules with the size adjustment + channel connection method of the features. "Retain and "Retain " respectively indicate and results.

[0138] Table 5 Impact of key components on performance in DSDM models compared on the Rain100H dataset

[0139] ,

[0140] Experimental results show that removing the L2P or P2L modules significantly degrades performance, indicating that the complementary information between the pixel and latent spaces during the diffusion process can effectively improve image restoration quality. While using splicing instead of L2P / P2L can preserve some information, the lack of an information regulation mechanism still degrades model performance. Directly using the output of pixel or latent space diffusion (without passing it through the FRN) significantly undermines image structural integrity and detail perception compared to the fused results. In summary, this complementary fusion mechanism between the pixel and latent spaces is the core reason our method outperforms traditional single-space diffusion models.

[0141] S4.4.2: Diffusion prediction target verification: Table 6 shows the influence of the MU-Net prediction target in DSDM during the back diffusion process. The prediction target in Table 6 is When, through Calculated .

[0142] During the training process, the true value image or the true value potential feature is first predicted, and then the degraded image or the potential feature of the degraded image is subtracted in MU-Net to obtain and , to replace the RDDM lieutenant general and The intuitive motivation for using the inverse diffusion prediction target directly is to ensure that the target distributions of different types of degraded image mappings are the same at each step of the inverse diffusion process, thereby alleviating the training conflicts caused by assigning different mapping distributions to different degradation tasks. As shown in Table 6, this simple modification improves the performance of DSDM on each image restoration task.

[0143] Table 6 Diffusion prediction target verification The impact of MU-Net prediction target in DSDM during back diffusion

[0144] ,

[0145] S4.4.3: Diffusion step analysis: e.g. Figure 10 As shown, Figure 10 This study demonstrates the impact of the number of diffusion steps, T, on MU-Net model performance and inference time. Performance reaches peak value when T = 3. Further increasing T offers little performance improvement but incurs higher computational overhead. Based on this ablation study, T = 3 is chosen in the DSDM model for general image restoration tasks to balance performance and efficiency.

[0146] S4.4.4: Model Efficiency Analysis: Table 7 compares the memory usage, FLOPS, and model inference time of various common image restoration methods. Compared with other existing methods, the proposed method is competitive in terms of memory consumption, computational FLOPS, and inference time.

[0147] Table 7 Model complexity analysis

[0148] . Example 2:

[0149] A universal image restoration system for dual-space joint diffusion, used to implement the universal image restoration method for dual-space joint diffusion described in Example 1, comprising:

[0150] Model building module: used to build a dual-space diffusion model DSDM with degraded images as input and true value images as output;

[0151] Model training module: used to train the dual-space diffusion model DSDM using image datasets;

[0152] Image restoration module: used to restore the collected degraded images through the trained model;

[0153] The dual-space diffusion model DSDM includes a MU-Net network and a fusion refinement network FRN; the MU-Net network includes a first and a second U-net convolutional neural network that perform diffusion processes in pixel space and latent space, respectively, and the fusion refinement network FRN is used to fuse the outputs of the first and second U-net convolutional neural networks and output a restored image.

[0154] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered by the scope of protection of the claims of the present invention.

Claims

1. A general image restoration method based on dual-space joint diffusion, characterized in that: The specific steps are: S1: Construct a dual-space diffusion model DSDM with degraded image as input and true image as output; S2: Use the image dataset to train the dual-space diffusion model DSDM; S3: Restore the collected degraded image using the trained model; The dual-space diffusion model DSDM includes a MU-Net network and a fusion refinement network FRN; the MU-Net network includes a first and a second U-net convolutional neural network that perform diffusion processes in pixel space and latent space, respectively, and the fusion refinement network FRN is used to fuse the outputs of the first and second U-net convolutional neural networks and output a restored image; The first and second U-net convolutional neural networks respectively include a pixel feature enhancement module L2P based on potential features and a potential feature enhancement module P2L based on pixel features; The pixel feature enhancement module L2P based on latent features is used to select information from the back diffusion process of the latent space to enhance the pixel space back diffusion process. The latent feature enhancement module P2L based on pixel features is used to select information from the back diffusion process of the pixel space to enhance the latent space back diffusion process. The specific method of the pixel feature enhancement module L2P based on the latent feature to select information from the latent space back diffusion process to enhance the pixel space back diffusion process is as follows: Pixel spatial features of the k-th layer decoder in the first and second U-net convolutional neural networks and latent space features Normalization is performed and the normalized pixel space features are Perform downsampling to obtain downsampled features ; Normalize the latent space features and downsampled features Perform channel splicing to form preliminary fusion features, and then extract the preliminary fusion features through two layers of convolutional layers and corresponding activation functions to generate filter masks ; Normalize the latent space features With filter mask Perform channel splicing and upsampling to obtain spliced ​​upsampled features, and then use Swin Transformer blocks and convolution layers to combine spliced ​​upsampled features and normalized pixel space features. Extract and compare the extracted features with the normalized pixel space features Fusion to obtain enhanced pixel space features ; The specific method of the pixel feature-based latent feature enhancement module P2L to select information from the reverse diffusion process of the pixel space to enhance the potential space reverse diffusion process is as follows: Pixel spatial features of the k-th layer decoder in the first and second U-net convolutional neural networks and latent space features Normalize and normalize the latent space features Perform upsampling to obtain upsampled features ; Normalize pixel space features and upsampled features Perform channel splicing to form preliminary fusion features, then extract the preliminary fusion features through the convolution layer and the corresponding activation function to generate the filter mask ; Normalize pixel space features With filter mask Perform channel splicing and downsampling to obtain spliced ​​downsampled features, and then use the self-attention mechanism block and convolution layer to combine the spliced ​​downsampled features and normalized latent space features. Extract and compare the extracted features with the normalized latent space features Fusion to obtain enhanced pixel space features .

2. The universal image restoration method of dual-space joint diffusion according to claim 1, characterized in that: Both the pixel space diffusion and the latent space diffusion of the degraded image include a forward diffusion process and a backward diffusion process, wherein the backward diffusion process takes the output of the forward diffusion process as input.

3. The universal image restoration method of dual-space joint diffusion according to claim 1, characterized in that: The specific method of the fusion refinement network FRN fusing the outputs of the first and second U-net convolutional neural networks and outputting the restored image is as follows: The outputs of the first and second U-net convolutional neural networks and Perform channel splicing and extract hierarchical features through convolutional layers and channel splitting modules; perform convolution or dilated convolution operations on the hierarchical features, and perform residual operations on the features after the operation to obtain multi-scale hierarchical features; The multi-scale hierarchical features are channel-spliced, and then the channel-spliced ​​multi-scale hierarchical features are fused through the convolution layer, sliding window attention block, residual block and convolution layer to generate the final image.

4. The universal image restoration method of dual-space joint diffusion according to claim 2, characterized in that: The specific method of training the dual-space diffusion model DSDM using image datasets is as follows: S2.1: Construct the loss functions of pixel space diffusion and latent space diffusion respectively, and use the sum of the loss functions of pixel space diffusion and latent space diffusion as the loss function of the MU-Net network; S2.2: Train the MU-Net network using the image dataset as the training set data to obtain the trained MU-Net network; S2.3: Freeze the trained MU-Net network parameters and perform T-step back diffusion to obtain the recovery results in two different diffusion spaces; S2.4: Construct a loss function for the fusion and refinement network FRN, train the fusion and refinement network FRN with the recovery results of two different diffusion spaces as input, and obtain the trained fusion and refinement network FRN.

5. The universal image restoration method of dual-space joint diffusion according to claim 2, characterized in that: The collected degraded image is restored using the trained model. The specific steps are as follows: S3.1: Input the collected degraded image into the trained dual-space diffusion model DSDM and perform forward diffusion in pixel space and latent space respectively; S3.2: Input the forward diffusion outputs in pixel space and latent space into the first and second U-net convolutional neural networks respectively to perform backward diffusion; S3.3: The outputs of the first and second U-net convolutional neural networks are used as inputs of the fusion refinement network FRN, the outputs of the first and second U-net convolutional neural networks are fused through the fusion refinement network FRN, and a restored image is output.

6. A universal image restoration system based on dual-space joint diffusion, characterized in that: A general image restoration method for implementing the dual-space joint diffusion method according to any one of claims 1 to 5, comprising: Model building module: used to build a dual-space diffusion model DSDM with degraded images as input and true value images as output; Model training module: used to train the dual-space diffusion model DSDM using image datasets; Image restoration module: used to restore the collected degraded images through the trained model; The dual-space diffusion model DSDM includes a MU-Net network and a fusion refinement network FRN; the MU-Net network includes a first and a second U-net convolutional neural network that perform diffusion processes in pixel space and latent space, respectively, and the fusion refinement network FRN is used to fuse the outputs of the first and second U-net convolutional neural networks and output a restored image.

Images