A diffusion model processing method, apparatus, electronic device, and storage medium for unified image restoration.

Abstract

This invention discloses a diffusion model processing method, apparatus, electronic device, and storage medium for unified image restoration. Based on a virtual consistency function, this invention establishes a virtual consistency model and introduces a lightweight noise correction and cue word refinement network, which can be used for noise and cue embedding modification, enhancing image restoration performance without retraining the pre-trained diffusion model. By using the virtual consistency function, the virtual consistency model VCMUIR leverages the uniform degradation characteristics of the high-noise space of the diffusion model, effectively solving general image restoration problems while significantly improving fidelity. This invention can adaptively handle different degradation types to enhance robustness and can simultaneously handle different degradation tasks, thus providing a general image restoration model that eliminates the need for task-specific models, thereby improving image restoration efficiency. This invention has wide applications in the field of image processing technology.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a diffusion model processing method, apparatus, electronic device, and storage medium for unified image restoration. Background Technology

[0002] In low-light, foggy, rainy, snowy, or moving subjects, or a combination of these conditions, photographs often suffer from blurred subjects or severe noise interference, resulting in degraded images. Image restoration refers to processing degraded images to obtain clearer images with the same content. Currently, techniques such as AirNet, IDR, Painter, ProRes, and DA-CLIP can be applied to image restoration, but these techniques generally have limitations. For example, AirNet uses a module to map different degradation distributions to a common distribution through contrastive learning, but faces challenges in training and performance. IDR observes that degradation types can be segmented using singular value decomposition, and a clear image can be reconstructed by refactoring singular values ​​and vectors. Painter, ProRes, and DA-CLIP utilize dynamic learning to leverage the potential of large models for image restoration; however, due to their multi-part mapping strategies, methods like IDR, Painter, ProRes, and DA-CLIP only achieve modest performance improvements and require significant model complexity. Summary of the Invention

[0003] In view of the technical problems of limited performance and complex models in current image restoration technology, the purpose of this invention is to provide a diffusion model processing method, device, electronic device and storage medium for unified image restoration.

[0004] On one hand, embodiments of the present invention include a diffusion model processing method for unified image restoration, the diffusion model processing method for unified image restoration comprising the following steps:

[0005] Perform at least one iteration; each iteration includes the following steps:

[0006] Get the input image and prompt words;

[0007] The input image is subjected to noise processing to obtain a noisy image;

[0008] The noise-corrected image is processed using a noise correction network to obtain noise spatial features;

[0009] The noise spatial features and the cue words are processed using a pre-trained diffusion model to obtain noise residuals;

[0010] The input image is subjected to noise reconstruction processing to obtain reconstructed noise;

[0011] The virtual consistency function is used to process the noise spatial features, the noise residuals, and the reconstructed noise to obtain the restored image of the iterative process in this round.

[0012] Furthermore, the step of processing the noise spatial features and the cue words using a pre-trained diffusion model to obtain noise residuals includes:

[0013] Obtain the refined embedding features corresponding to the prompt words;

[0014] The noise spatial features and the refined embedding features are input into the pre-trained diffusion model for processing;

[0015] Obtain the first noise prediction information and the second noise prediction information obtained by processing the pre-trained diffusion model;

[0016] The noise residual is obtained by performing a difference operation on the first noise prediction information and the second noise prediction information.

[0017] Further, obtaining the refined embedding features corresponding to the prompt word includes:

[0018] The prompt words are processed using a text encoder to obtain prompt word embedding features;

[0019] The input image is processed using a CLIP network to obtain CLIP embedding features;

[0020] The prompt word embedding features and the CLIP embedding features are subjected to prompt word refinement processing to obtain the refined embedding features.

[0021] Further, the step of processing the noise spatial features, the noise residual, and the reconstructed noise using a virtual consistency function to obtain the restored image of the current iteration includes:

[0022] According to the formula

[0023]

[0024] Perform calculations to obtain the restored image from the iterative process described in this round. in, Let t represent the spatial characteristics of the noise, and t represent the current iteration sequence. Let Δ be the reconstructed noise, and △ be the noise residual. For a low-quality image, f() satisfies

[0025]

[0026] in

[0027]

[0028] z0 represents the initial sampled data, z t This represents the result of z0 after t noise additions, ∈(z t ,t;z0) represents the noise predictor, α t The hyperparameters for the variance conforming to a Gaussian distribution.

[0029] Furthermore, when the current iteration process is the first iteration process, the input image is the image to be processed;

[0030] When the current iteration process is an iteration process other than the first iteration process, the input image is the restored image obtained by the previous iteration process.

[0031] Furthermore, when the diffusion model processing method for unified image restoration is executed during the training phase, each iteration process also includes the following steps:

[0032] Obtain the high-quality image corresponding to the input image;

[0033] The noise correction network is trained under supervision based on the high-quality image and the restored image.

[0034] Furthermore, when the diffusion model processing method for unified image restoration is executed during the inference phase, the diffusion model processing method for unified image restoration further includes the following steps:

[0035] Obtain the image enhancement level;

[0036] The total number of rounds is determined based on the image enhancement level;

[0037] After completing any round of the iteration process, the number of rounds executed in the iteration process is counted. If the number of rounds executed is less than the total number of rounds, the next round of the iteration process is executed; otherwise, the execution of all the iteration processes is terminated.

[0038] Obtain the restored image obtained from the last round of the iterative process.

[0039] On the other hand, embodiments of the present invention also include a computer device, including a memory and a processor, the memory for storing at least one program, and the processor for loading at least one program to execute the diffusion model processing method for unified image restoration in the embodiments.

[0040] On the other hand, embodiments of the present invention also include an electronic device, the electronic device including a processor and a memory, the memory storing at least one instruction, at least one program, code set or instruction set, the at least one instruction, the at least one program, the code set or instruction set being loaded and executed by the processor to implement the diffusion model processing method for unified image restoration in the embodiments.

[0041] On the other hand, embodiments of the present invention also include a computer-readable storage medium storing a processor-executable program, which, when executed by a processor, is used to perform the diffusion model processing method for unified image restoration in the embodiments.

[0042] The beneficial effects of this invention are as follows: The diffusion model processing method for unified image restoration in the embodiments establishes a virtual consistency model based on a virtual consistency function. Specifically, a lightweight noise correction and cue word refinement network is introduced, which can be used for noise and cue embedding modification, enhancing the performance of image restoration without retraining the pre-trained diffusion model. By using the virtual consistency function, the virtual consistency model VCMUIR utilizes the uniform degradation characteristics of the high-noise space of the diffusion model, effectively solving general image restoration problems and significantly improving fidelity. Moreover, the virtual consistency model VCMUIR is a unified development model, and the noise correction network can be trained during the training phase, thereby adaptively handling different degradation types to enhance robustness. It can handle different degradation tasks simultaneously, thus being a general image restoration model, eliminating the need for specialized models for specific tasks, and thus improving image restoration efficiency. Attached Figure Description

[0043] Figure 1 This is a schematic diagram illustrating the principle of the virtual consistency model in the embodiment;

[0044] Figure 2 This is a schematic diagram illustrating the overall steps of the diffusion model processing method for unified image restoration in the embodiment.

[0045] Figure 3 This is a schematic diagram of the virtual consistency model in the embodiment;

[0046] Figure 4 This is a schematic diagram of the prompt word refinement network in the embodiment;

[0047] Figure 5 This is a schematic diagram illustrating the steps of the diffusion model processing method used for unified image restoration during the training phase in this embodiment.

[0048] Figure 6This is a schematic diagram illustrating the steps of the diffusion model processing method used for unified image restoration during the inference stage in this embodiment. Detailed Implementation

[0049] To address the performance limitations and model complexity of current image restoration techniques, DiffUIR proposes a selective hourglass mapping strategy based on a residual diffusion model, achieving both shared distribution mapping and strong conditional guidance. However, DiffUIR heavily relies on the residual diffusion model, making it difficult to leverage the strong priors found in community-based diffusion models, thus hindering further improvements in general image restoration. Furthermore, the original training objective of the diffusion model is not to directly solve the image restoration problem, which may lead to significant fidelity loss.

[0050] Therefore, a key challenge in image restoration technology is how to effectively learn multiple degradation distributions simultaneously to enhance the performance of image restoration tasks with various degradation types. The standard diffusion model exhibits degradation uniformity and significant content in high-noise environments, making it well-suited for general image restoration.

[0051] The forward propagation and denoising processes of the standard diffusion model are as follows:

[0052]

[0053] Based on the standard diffusion model, a key consistency function f(z) is introduced. t By using a key consistency function to ensure consistent outputs at all points on the probability flow ordinary differential equation (ODE), a consistency model can be obtained. Compared to the standard diffusion model, the consistency model significantly accelerates the generation process. In the consistency model, sampling occurs over a series of time steps T. 1:n Formed on [t0,T]. Based on initial noise. and Beginning at each time step τ i The sampling process for ∈~N(0,1) and the iterative update multi-step consistent sampling process are as follows:

[0054]

[0055] Consider a special case of formula (2), when σ t Take at all time steps t When, formula (2) can be rewritten as

[0056]

[0057] Let ∈ represent a more general noise predictor, then we can obtain

[0058]

[0059] Combining formulas (5) and (6), a sampling process similar to multi-step consistency sampling [formulas (3) and (4)] is as follows:

[0060]

[0061] To ensure f(z) t The expression (z,t;z0) is self-constrained and can be viewed as a consistency function. The objective of DDCM is f(z,t;z0). t (t; z0) = z0. This involves directly solving the equation to calculate ∈ without parameterization, as shown below:

[0062]

[0063] In the formula ∈ rec To reconstruct the noise z0.

[0064] The consistency function f(z) obtained by formula (6) t The expression f(z,t;z0) is not derived from the probability flow ordinary differential equation, and the consistency function f(z) can be used to express the expression. t The function ,t;z0) is called the virtual consistency function, and the model that uses it is called a virtual consistency model (VCM). A new virtual consistency model (essentially a diffusion model) for general image restoration can be designed based on formulas (7) and (8), called the Virtual Consistency Model for Image Restoration (VCMUIR). In this embodiment, the principle of data processing by the virtual consistency model (VCMUIR) is as follows: Figure 1 As shown, the consensus function used by the Virtual Consistency Model (VCMUIR) is as follows:

[0065]

[0066] In the formula ∈ θ (z t ,t,c * ) represents a pre-trained diffusion model; c * The input conditions (specifically, the refined embedded features c) HQ and c LQ ); Represents a high-quality image for prediction; This represents a low-quality image, equivalent to the high-quality image output from the previous step, initially set to... It is based on formula (1) The result of adding noise. When t=0, due to... and They can cancel each other out; substituting them into the equation yields... Therefore, equation (9.1) satisfies the boundary conditions of the consistency model. Through this objective, the sampling process of VCMUIR can be... Figure 1 See section (b) of the diagram. Noise correction network r φ The actual aim is to transform low-quality images It was re-identified as a high-quality low-volume component in a high-noise space.

[0067] Based on the above principles, this embodiment provides a diffusion model processing method for unified image restoration. (Refer to...) Figure 2 The diffusion model processing method for unified image restoration includes the following steps:

[0068] S1. Obtain the image to be processed and the prompt words;

[0069] Perform at least one iteration; each iteration includes the following steps:

[0070] S201. Obtain the input image and prompt words;

[0071] S202. Add noise to the input image to obtain a noisy image;

[0072] S203. Use a noise correction network to process the noisy image and obtain the spatial features of the noise;

[0073] S204. Use a pre-trained diffusion model to process the spatial features of noise and cue words to obtain noise residuals;

[0074] S205. Perform noise reconstruction processing on the input image to obtain the reconstructed noise;

[0075] S206. Use the virtual consistency function to process the noise spatial features, noise residuals and reconstruction noise to obtain the restored image of this iteration process.

[0076] In this embodiment, the image to be processed obtained in step S1 can be a low-quality image (degraded image) or a high-quality image. A low-quality image is an image captured in low-light, foggy, rainy, snowy, or moving scenes, or an image obtained by artificially adding noise; such images have low-quality characteristics such as blurriness. A high-quality image is an image with less noise and clearer content compared to a low-quality image. The prompt words are text used to describe the quality of the image to be processed. For example, if there are low-quality areas in the image, the prompt words can include low-quality image prompt words, such as "fog," "rain," "snow," "low light," "blur," "motion blur," "low resolution," "unnatural," "low quality," etc. If there are high-quality areas in the image, the prompt words can include high-quality image prompt words, such as "clean," "realistic," "high resolution," "natural," "high quality," "refined details," etc.

[0077] Steps S201-S206 constitute one iteration process, and generally multiple iterations are required. The t-th iteration is used as an example. The processing flow executed by the virtual consensus model in the t-th iteration is as follows: Figure 3 As shown. Specifically, Figure 3 The encoder, CLIP, cue word refinement network, noise correction network, and diffusion model (main network) constitute the virtual consistency model. (Refer to...) Figure 3 The virtual consistency model can also include networks such as decoders and LoRA adapters.

[0078] If the current iteration, i.e., the t-th iteration, is the 1st iteration, then the input image to be obtained in step S201 is the image to be processed obtained in step S1. If the current iteration, i.e., the t-th iteration, is an iteration after the 1st iteration, then the input image to be obtained in step S201 is the restored image obtained in the previous iteration. The prompt words obtained in step S201 can be the same as the prompt words obtained in step S1.

[0079] In step S202, refer to Figure 3 The input image can be The input is fed into the encoder for encoding, thus obtaining the encoded image during the t-th iteration. Next, the coded image is processed using the principle of formula (1). Noise is added to obtain the noisy image during the t-th iteration. Specifically, for formula (1), that is...

[0080]

[0081] It can make one of them z is calculated by applying formula (1) t Then take Thus, a noisy image is obtained.

[0082] In step S203, refer to Figure 3 Add noise to the image The input is processed by a noise correction network to obtain the spatial features of the noise. Specifically, in this embodiment, a UNet network can be used as the noise correction network r. φ The noise correction network specifically applies the first formula in formula (9.2), which is...

[0083]

[0084] For noisy images Processing is performed to obtain the spatial characteristics of the noise. Where, r φ () indicates the processing procedure of the UNet network.

[0085] In this embodiment, when performing step S204, which is to process the noise spatial features and prompt words using the pre-trained diffusion model to obtain the noise residual, the following steps can be performed:

[0086] S20401. Obtain the refined embedding features corresponding to the prompt words;

[0087] S20402. Input the noise spatial features and refined embedding features into the pre-trained diffusion model for processing;

[0088] S20403. Obtain the first noise prediction information and the second noise prediction information obtained after processing by the pre-trained diffusion model;

[0089] S20404. Perform a difference calculation based on the first noise prediction information and the second noise prediction information to obtain the noise residual.

[0090] In step S20401, refer to Figure 3 The system can input prompt words (specifically, low-quality or high-quality image prompt words) into a text encoder for encoding processing to obtain the prompt word embedding features output by the text encoder; on the other hand, the input image can be input into a CLIP network for feature extraction processing to obtain CLIP embedding features; the prompt word embedding features and CLIP embedding features can be input into a prompt word thinning network for processing.

[0091] In this embodiment, the structure of the cue word refinement network is as follows: Figure 4 As shown, it includes a cross-attention module, a CLIP feature mapping module, and a Transformer decoder. The structures of the cross-attention module and the CLIP feature mapping module are also... Figure 4 As shown in the diagram, the cross-attention module processes the cue word embedding features, the CLIP feature mapping module processes the CLIP embedding features, and the final results are summed to obtain the refined embedding feature c. * (Specifically, for low-quality image cue words, the refined embedding features obtained can be represented as c) LQ For high-quality image cues, the refined embedding features can be represented as c. HQ The CLIP network extracts image features and initial quality descriptive cue embeddings from the input image. These inputs are then processed by cross-attention and converter decoders in the cue refinement network to generate refined cue embeddings, i.e., refinement of the embedding features c. LQ and c HQ .

[0092] In step S20402, the noise spatial features obtained in step S203 are executed. And the refined embedding features c obtained from step SS20401 * These are combined and input into a pre-trained diffusion model for processing. The pre-trained diffusion model specifically applies the second formula in equation (9.2), which is...

[0093]

[0094] Processing is performed. Specifically, ∈ θ () indicates the processing steps of the pre-trained diffusion model. In step S20403, the pre-trained diffusion model applies high-quality image prompts to the word c. HQ The first noise prediction information obtained after processing In step S20403, the pre-trained diffusion model detects low-quality image prompts c. LQ The second noise prediction information obtained through processing. △∈ is the result of step S20404 on the first noise prediction information. Second noise prediction information The noise residual is obtained by subtracting the values.

[0095] In step S205, refer to Figure 4 For the input image (corresponding encoded image) Noise reconstruction processing is performed to obtain the reconstructed noise. Specifically, the third formula in formula (9.2) is used, that is...

[0096]

[0097] Based on the input image (corresponding encoded image) ) and noise spatial characteristics Calculations are performed to obtain the reconstruction noise during the t-th iteration.

[0098] In this embodiment, when performing step S206, which is to process the noise spatial features, noise residuals, and reconstruction noise using a virtual consistency function to obtain the restored image of the current iteration process, the following steps can be specifically performed:

[0099] According to formula 9.1, that is

[0100]

[0101] Perform the calculation to obtain the restored image of the current iteration process, which is also the t-th iteration process. in, The noise spatial features obtained in step S203 are as follows. The reconstructed noise is obtained by performing step S205, and Δ∈ is the noise residual obtained by performing step S204.

[0102] In this embodiment, f() in formula 9.1 satisfies

[0103]

[0104] in

[0105]

[0106] z0 represents the initial sampled data, z t This represents the result of z0 after t noise additions, ∈(z t ,t;z0) represents the noise predictor, α t This is a hyperparameter that conforms to the Gaussian distribution variance. Specifically, during the calculation, we can let the variance in f() be... The calculation is performed to obtain the restored image of the t-th iteration process.

[0107] If the iteration termination condition is not met after the t-th iteration, then the (t+1)-th iteration will be executed. The restored image from the t-th iteration... The relatively high-quality image will become the input image for the (t+1)th iteration process, while the relatively low-quality image will be processed in the new iteration process.

[0108] In this embodiment, Figure 2 and Figure 3The process of a diffusion model for general image restoration, which processes the image to be processed and the prompt words, is illustrated. A training phase can be performed to train the diffusion model for general image restoration before applying it to inference (to restore degraded images that actually require restoration).

[0109] During the training phase, refer to Figure 5 The image to be processed obtained in step S1 is specifically a sample image. The sample image can be an image taken in a specially set specific scene (e.g., low light) or an image obtained by deliberately adding noise or other interference factors. During step S1, a high-quality image corresponding to the sample image is obtained. The high-quality image has the same content (the subject being photographed) as the sample image, but was taken in a better shooting scene (e.g., sufficient light, sunny weather, and the subject being photographed still), or has already undergone noise reduction processing.

[0110] During the training phase, after each iteration (e.g., the t-th iteration), the noise correction network r is adjusted based on the high-quality image corresponding to the sample image and the restored image obtained in the t-th iteration. φ Conduct supervised training.

[0111] In this embodiment, the noise correction network r φ The training needs to meet two main requirements: (1) Implement the recovery function r in a high-noise space. φ (2) Provide image restoration supervision in image space.

[0112] Regarding the first requirement, due to the noisy image... It is a noise correction network r φ The input, when t is large enough, r φ The range is naturally in the high-noise space. Taking advantage of uniform degradation, a simple and lightweight UNet network can be used to correct input noise. For the second condition, the optimization objective is first defined. By expanding the consistency function [Equation (9)], we can obtain

[0113]

[0114] Equation (10) shows that the noise correction network r φ The main focus is fitting the noise residuals. When the pre-trained diffusion model is a consistent model, it can be assumed that these residuals are approximately consistent across different time steps. Therefore, a fixed total step size (the maximum value of t) can be chosen, and the perceptual loss function can be determined using L1 loss and LPIPS as follows.

[0115]

[0116] in The restored image obtained in the t-th iteration process λ represents the high-quality image corresponding to the sample image. λ is the balance coefficient. The time step is set to 999 (i.e., the maximum value of t, which is the total number of iterations performed is 999) to ensure maximum uniform degradation. Although only one optimization step is taken, it is sufficient to generalize to a multi-step denoising process. Obviously, Equation (11) satisfies the conditions for supervision in the image space.

[0117] Reference Figure 5 In the t-th iteration, after executing steps S201-S206, the loss function value can be calculated using formula (11). Based on the loss function value, the noise correction network r is adjusted. φ Supervised training is performed. Due to the noise correction network r in this embodiment... φ It is a Unet network, therefore the training methods for Unet networks can be used to train the noise correction network r. φ Conduct training.

[0118] Reference Figure 3 In this embodiment, the encoder, CLIP, text encoder, pre-trained diffusion network, and decoder are frozen, therefore according to Figure 5 When supervising the training of the process, it is actually possible to train only the noise correction network r. φ Training, or training the noise correction network r φ The network was trained using a refinement method based on prompt words.

[0119] pass Figure 5 The steps shown demonstrate that the diffusion model processing method for unified image restoration can be used to train VCMUIR.

[0120] After training the diffusion model for general image restoration, the diffusion model for general image restoration can be applied to the inference stage to perform image restoration.

[0121] During the reasoning phase, refer to Figure 6 The image to be processed obtained in step S1 is a degraded image. The degraded image may be an image taken under poor photographic conditions or an image affected by noise or other interference factors.

[0122] Equation (10) proves that VCMUIR actually learns the noise residual between the degraded image and the high-quality image in a high-noise space. Therefore, through multi-step inference, the image enhancement level can be adjusted to meet the preferences of different users. During the inference phase, the image enhancement level can be determined according to user preferences, where the image enhancement level represents the degree to which the user wants to enhance the degraded image. The greater the degree of enhancement the user wants to give to the degraded image, the higher the image enhancement level, and the larger the total number of rounds determined accordingly.

[0123] Reference Figure 6 After each iteration, the system checks if the total number of iterations has been reached. If not, the restored image from the last iteration is used as the input for the next iteration. If the total number of iterations has been reached, the system terminates all iterations and outputs the restored image from the last iteration. Figure 3 The restored image is input into the decoder for decoding, thereby obtaining the restored result of the degraded image.

[0124] Reference Figure 3 Furthermore, trainable LoRA and skip connections can be added to the decoder that processes the restored image, thereby enhancing the detail restoration of the restored result.

[0125] pass Figure 6 The steps shown demonstrate that the diffusion model processing method for unified image restoration can be applied to the following scenarios:

[0126] 1. Enhance image brightness in low-light scenes

[0127] 2. Image dehazing in foggy scenes

[0128] 3. Remove rain from images in rainy scenes

[0129] 4. Remove snow from images with snow scenes

[0130] 5. Image Deblurring in Motion-Blurred Scenes

[0131] 6. Image Enhancement in Hybrid Degradation Scenes

[0132] This invention constructs a method called VCMUIR for image restoration in a high-noise space, proposing a virtual consistency model for general image restoration. This model effectively achieves general image restoration, ensuring accurate supervision of the image in the image space through the proposed consistency function. Degraded images exhibit uniform degradation characteristics in the high-noise space of the diffusion model, thus resolving degradation conflict issues. Simultaneously, ensuring accurate supervision data during training is crucial for image restoration, and the image space is the optimal space for achieving this. Extensive experiments demonstrate that this invention outperforms existing methods under 22 different degrees of degradation, including general restoration and challenging real-world restoration tasks.

[0133] The diffusion model processing method for unified image restoration in this embodiment re-engineers the inverse denoising process in the diffusion model by designing a virtual consistency function, thereby establishing a virtual consistency model VCMUIR. The virtual consistency model VCMUIR naturally unifies different degradations through a high-noise spatial standard diffusion model while preserving core image content. It addresses the performance limitations and model complexity issues of current image restoration techniques while maximizing the retention of prior knowledge from the pre-trained diffusion model. Specifically, the virtual consistency model VCMUIR in this embodiment introduces a lightweight noise correction and cue word refinement network, which can be used for noise and cue embedding modification, enhancing image restoration performance without retraining the pre-trained diffusion model. By using the virtual consistency function, the virtual consistency model VCM... UIR leverages the uniform degradation characteristics of the high-noise space in the diffusion model, effectively solving general image restoration problems while significantly improving fidelity. The Virtual Consistency Model (VCMUIR) is a unified development model that can train its noise correction network during the training phase, thereby adaptively handling different degradation types to enhance robustness. It can handle different degradation tasks simultaneously, making it a general image restoration model that eliminates the need for task-specific models, thus improving image restoration efficiency. The image restoration and supervision functions of VCMUIR are decoupled. It can enhance detail restoration by freezing the pre-trained diffusion model, introducing a lightweight noise correction network, a cue word thinning network, and adding trainable LoRA and skip connections to the decoder, achieving image restoration for various degradation types.

[0134] A computer program for executing the diffusion model processing method for unified image restoration in this embodiment can be written into a computer device or storage medium. When the computer program is read out and run, the diffusion model processing method for unified image restoration in this embodiment is executed, thereby achieving the same technical effect as the diffusion model processing method for unified image restoration in the embodiment.

[0135] It should be noted that, unless otherwise specified, when a feature is referred to as "fixed" or "connected" to another feature, it can be directly fixed or connected to the other feature, or indirectly fixed or connected to the other feature. Furthermore, the descriptions of "upper," "lower," "left," and "right" used in this disclosure are only relative to the relative positional relationships of the components of this disclosure in the accompanying drawings. The singular forms "a," "an," and "the" used in this disclosure are also intended to include the plural forms, unless the context clearly indicates otherwise. Moreover, unless otherwise defined, all technical and scientific terms used in this embodiment have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in this embodiment specification is only for describing particular embodiments and is not intended to limit the invention. The term "and / or" as used in this embodiment includes any combination of one or more of the associated listed items.

[0136] It should be understood that although the terms first, second, third, etc., may be used to describe various elements in this disclosure, these elements should not be limited to these terms. These terms are only used to distinguish elements of the same type from each other. For example, a first element may also be referred to as a second element without departing from the scope of this disclosure, and similarly, a second element may also be referred to as a first element. The use of any and all instances or exemplary language (“e.g.,” “such as,” etc.) provided in this embodiment is intended only to better illustrate embodiments of the invention and, unless otherwise required, does not impose a limitation on the scope of the invention.

[0137] It should be recognized that embodiments of the present invention can be implemented or carried out by computer hardware, a combination of hardware and software, or by computer instructions stored in a non-transitory computer-readable storage medium. The method can be implemented using standard programming techniques—including a non-transitory computer-readable storage medium configured with a computer program, wherein such a storage medium causes the computer to operate in a specific and predefined manner—according to the methods and drawings described in the specific embodiments. Each program can be implemented in a high-level procedural or object-oriented programming language to communicate with the computer system. However, if desired, the program can be implemented in assembly or machine language. In any case, the language can be a compiled or interpreted language. Furthermore, for this purpose, the program can run on a programmed application-specific integrated circuit (ASIC).

[0138] Furthermore, the procedures described in this embodiment can be performed in any suitable order unless otherwise indicated by this embodiment or clearly contradicted by the context. The procedures (or variations and / or combinations thereof) described in this embodiment can be executed under the control of one or more computer systems configured with executable instructions, and can be implemented by hardware or a combination thereof as code (e.g., executable instructions, one or more computer programs, or one or more applications) that commonly executes on one or more processors. A computer program includes multiple instructions executable by one or more processors.

[0139] Furthermore, the method can be implemented in any suitable type of computing platform, including but not limited to personal computers, minicomputers, mainframes, workstations, networked or distributed computing environments, standalone or integrated computer platforms, or in communication with charged particle tools or other imaging devices, etc. Aspects of the invention can be implemented as machine-readable code stored on a non-transitory storage medium or device, whether removable or integrated into a computing platform, such as a hard disk, optical read and / or write storage medium, RAM, ROM, etc., such that it is readable by a programmable computer, and when the storage medium or device is read by the computer, it can be used to configure and operate the computer to perform the processes described herein. Furthermore, the machine-readable code, or portions thereof, can be transmitted via wired or wireless networks. The invention of this embodiment includes these and other different types of non-transitory computer-readable storage media when such media comprises instructions or programs that implement the steps above in conjunction with a microprocessor or other data processor. When programmed according to the methods and techniques of the invention, the invention also includes the computer itself.

[0140] A computer program can be applied to input data to perform the functions of this embodiment, thereby transforming the input data to generate output data stored in non-volatile memory. The output information can also be applied to one or more output devices, such as a display. In a preferred embodiment of the invention, the transformed data represents physical and tangible objects, including specific visual depictions of physical and tangible objects generated on the display.

[0141] The above are merely preferred embodiments of the present invention. The present invention is not limited to the above-described embodiments. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention, as long as they achieve the technical effects of the present invention by the same means, should be included within the scope of protection of the present invention. Within the scope of protection of the present invention, the technical solutions and / or implementation methods can have various modifications and variations.

Claims

1. A diffusion model processing method for unified image restoration, characterized in that, The diffusion model processing method for unified image restoration includes: Perform at least one iteration; each iteration includes the following steps: Get the input image and prompt words; The input image is subjected to noise processing to obtain a noisy image; The noise-correcting network is used to process the noisy image to obtain noise spatial features; The noise spatial features and the cue words are processed using a pre-trained diffusion model to obtain noise residuals; The input image is subjected to noise reconstruction processing to obtain reconstructed noise; The virtual consistency function is used to process the noise spatial features, the noise residuals, and the reconstructed noise to obtain the restored image of the iteration process in this round; The step of processing the noise spatial features, the noise residual, and the reconstructed noise using a virtual consistency function to obtain the restored image of the current iteration includes: According to the formula Perform calculations to obtain the restored image from the iterative process described in this round. ;in, The noise spatial characteristics, This indicates the round order of the current iteration process. For the reconstructed noise, The noise residual, Indicates a low-quality image. satisfy in This represents the initial sampled data. express The result after t noise additions Indicates a noise predictor. The hyperparameters for the variance conforming to a Gaussian distribution.

2. The diffusion model processing method for unified image restoration according to claim 1, characterized in that, The step of processing the noise spatial features and the cue words using a pre-trained diffusion model to obtain noise residuals includes: Obtain the refined embedding features corresponding to the prompt words; The noise spatial features and the refined embedding features are input into the pre-trained diffusion model for processing; Obtain the first noise prediction information and the second noise prediction information obtained by processing the pre-trained diffusion model; The noise residual is obtained by performing a difference operation on the first noise prediction information and the second noise prediction information.

3. The diffusion model processing method for unified image restoration according to claim 2, characterized in that, The step of obtaining the refined embedding features corresponding to the prompt words includes: The prompt words are processed using a text encoder to obtain prompt word embedding features; The input image is processed using a CLIP network to obtain CLIP embedding features; The prompt word embedding features and the CLIP embedding features are subjected to prompt word refinement processing to obtain the refined embedding features.

4. The diffusion model processing method for unified image restoration according to any one of claims 1-3, characterized in that: When the current iteration process is the first iteration process, the input image is the image to be processed; When the current iteration process is an iteration process other than the first iteration process, the input image is the restored image obtained by the previous iteration process.

5. The diffusion model processing method for unified image restoration according to any one of claims 1-3, characterized in that, When the diffusion model processing method for unified image restoration is executed during the training phase, each iteration process further includes the following steps: Obtain the high-quality image corresponding to the input image; The noise correction network is trained under supervision based on the high-quality image and the restored image.

6. The diffusion model processing method for unified image restoration according to any one of claims 1-3, characterized in that, When the diffusion model processing method for unified image restoration is executed during the inference phase, the diffusion model processing method for unified image restoration further includes the following steps: Obtain the image enhancement level; The total number of rounds is determined based on the image enhancement level; After completing any round of the iteration process, the number of rounds executed in the iteration process is counted. If the number of rounds executed is less than the total number of rounds, the next round of the iteration process is executed; otherwise, the execution of all the iteration processes is terminated. Obtain the restored image obtained from the last round of the iterative process.

7. A computer device, characterized in that, It includes a memory and a processor, the memory being used to store at least one program, and the processor being used to load at least one program to execute the diffusion model processing method for unified image restoration as described in any one of claims 1-6.

8. An electronic device, characterized in that, The electronic device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set, or an instruction set, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the method as described in any one of claims 1-6.

9. A computer-readable storage medium storing a processor-executable program, characterized in that, The processor-executable program, when executed by the processor, is used to perform the diffusion model processing method for unified image restoration as described in any one of claims 1-6.

Images