Image restoration method based on text guidance and detail preservation diffusion model

Abstract

The embodiment of the invention provides an image restoration method based on text guidance and a detail preservation diffusion model, and belongs to the technical field of image processing, and the method specifically comprises the steps: obtaining a preliminary degraded image according to a clear image, encoding the clear image into a hidden space through an encoder, and adding noise to obtain a noise hidden variable; inputting the preliminary degraded image and the noise hidden variable as conditions into a conditional diffusion model to predict noise added in the tth step, calculating to obtain a hidden variable with clear details, and decoding a new degraded image through a decoder; re-inputting the new degraded image and the noise hidden variable as conditions into the conditional diffusion model to obtain final predicted noise, and calculating loss according to the final predicted noise and the preliminarily predicted noise to carry out mixed training on the conditional diffusion model; and storing the model weight of the trained conditional diffusion model, and finishing a target image restoration process by using a DDIM reasoning process. According to the scheme, the repairing efficiency and adaptability are improved.

Description

Technical Field

[0001] The disclosed embodiments relate to the field of image processing technology, and in particular to an image restoration method based on text guidance and detail-preserving diffusion model. Background Art

[0002] Restoring visual artifacts from low-quality images, such as old photographs affected by various distortions, remains a core challenge in computer vision and has yet to be fully addressed. Despite significant progress in data-driven methods in recent years, the field still faces two key challenges. First, generating high-fidelity restored images with vibrant colors, rich details, and a high degree of realism remains challenging. Second, in the task of restoring old photos, the control of color details at the target level remains an unresolved issue.

[0003] Traditional methods typically train specialized image restoration models from scratch to preserve as much image detail as possible. These methods often use low-quality images as additional input to constrain the output space. While these methods have achieved some success in tasks such as super-resolution and deblurring, they are often designed for specific types of image degradation and require training from scratch, which limits the model's generalization ability and makes it difficult to adapt to complex scenes with multiple unknown degradations.

[0004] At the same time, large-scale diffusion models have demonstrated excellent performance in image generation and text-to-image generation. Therefore, some studies have attempted to leverage the generative priors of diffusion models for image restoration, introducing constraints during the back-diffusion process to improve restoration quality. However, such methods typically rely on accurate prior knowledge to describe the image degradation process and require individual optimization for each image, greatly limiting their practicality. Therefore, current research based on diffusion models rarely addresses image restoration tasks that simultaneously handle multiple unknown degradations. This is especially true in the specific application scenario of old photo restoration, which has complex degradation patterns and still lacks efficient and universal solutions.

[0005] It can be seen that there is an urgent need for an image restoration method based on text guidance and detail-preserving diffusion model with high restoration efficiency and adaptability. Summary of the Invention

[0006] In view of this, the embodiments of the present disclosure provide an image restoration method based on text guidance and detail-preserving diffusion model, which at least partially solves the problems of poor restoration efficiency and adaptability in the prior art.

[0007] The present disclosure provides an image restoration method based on text guidance and detail-preserving diffusion model, including:

[0008] Step 1: Obtain a preliminary degraded image based on the clear image, and encode the clear image into the latent space through an encoder and add noise to obtain a noise latent variable;

[0009] Step 2: The initial degraded image and the noise latent variable are used as conditions and input into the conditional diffusion model to predict the noise added in step t. Then, the latent variable that retains clear details is calculated and decoded into a new degraded image through the decoder.

[0010] Step 3: Re-input the new degraded image and noise latent variables as conditions into the conditional diffusion model to obtain the final predicted noise. The loss is calculated based on the final predicted noise and the preliminary predicted noise to hybrid train the conditional diffusion model.

[0011] Step 4: Save the model weights of the trained conditional diffusion model and use the DDIM reasoning process to complete the target image restoration process.

[0012] According to a specific implementation of the embodiment of the present disclosure, step 1 specifically includes:

[0013] Step 1.1, define a degradation model to transform the clear image I hq Degrade to obtain the initial degraded image I lq , where the expression of the degradation modeling is

[0014] I lq =φ ω (I hq )

[0015] Among them, φ ω (·) represents various preset degradation modeling operations;

[0016] Step 1.2, use the preset autoencoder to transform the clear image I hq Mapped to the latent space variable z0, and then gradually noised the latent space variable z0. During the noise addition process, the hyperparameters time step t and α are set. t , with variance β t Gaussian noise of ∈(0,1) is added to the encoded latent space variable z0 to generate the noisy latent variable z t , where the expression of the latent space variable is the noise addition formula at time t is

[0017] z0=E(I hq );

[0018] The noise addition formula at time t is:

[0019]

[0020] Where E(·) represents the autoencoder, t∈{1,…,T},∈~N(0,I), represents the noise of the standard Gaussian distribution, α t =1-βt,

[0021] According to a specific implementation of the embodiment of the present disclosure, step 2 specifically includes:

[0022] Step 2.1, the initial degraded image I lq and noise latent variable z t As a condition, it is sent to the conditional diffusion model to predict the noise added in step t

[0023]

[0024] Among them, ∈ θ (·) is the conditional diffusion model, z t is the noise latent variable of the tth step, t is the denoising time step, and the initial degraded image I lq As a condition, P is the text condition corresponding to the clear image;

[0025] Step 2.2, using the noise ∈ predicted by the model θ (z t ,t,I lq ,P), the latent variables of the t-1 step of the preliminary prediction are obtained by calculation and latent variables that retain clear details

[0026]

[0027] Step 2.3, the latent variables with clear details will be retained Decode the new degraded image through the decoder

[0028]

[0029] in, is the decoder, which is used to retain the latent variables with clear details Decode to pixel space to get the new degraded image

[0030] According to a specific implementation of the embodiment of the present disclosure, before step 2.1, the method further includes:

[0031] Use the BLIP model to obtain the text conditions corresponding to the clear image

[0032] P=BLIP(I hq )

[0033] Here, BLIP(·) represents the BLIP model.

[0034] According to a specific implementation of the embodiment of the present disclosure, step 3 specifically includes:

[0035] Step 3.1, based on the new degraded image and noise latent variable z t As a condition, it is re-sent into the conditional diffusion model to obtain the final predicted noise ∈ t ;

[0036] Step 3.2, using the noise ∈ predicted by the model t , the final predicted hidden variable z of step t-1 is obtained by calculation t-1

[0037]

[0038] Step 3.3, calculate the final predicted noise ∈ t The mean square error between the actual noise ∈ is used as the first loss

[0039]

[0040] Among them, E represents the mathematical expectation of different sampling variables, that is, taking the average value in multiple samples, U(1,T) represents the uniform distribution on the set 1,…,T, N(0,I) represents the multidimensional standard Gaussian distribution with mean 0 and covariance matrix as the unit matrix I, Represents the square of the L2 norm, that is, the square of the Euclidean distance, which is used to measure the gap between the predicted noise and the actual noise;

[0041] Step 3.4, calculate the noise of the initial prediction The mean square error between the actual noise ∈ is used as the second loss

[0042]

[0043] In step 3.5, the conditional diffusion model is trained jointly based on the first loss and the second loss.

[0044] According to a specific implementation of the embodiment of the present disclosure, step 4 specifically includes:

[0045] Noise from a standard Gaussian distribution A noise latent variable z is sampled from t and the corresponding preliminary degraded image I lq , for different time steps sampled by DDIM, the trained conditional diffusion model ∈ θ The model weights are used to predict the noise ∈ at each time step t θ (zt ,t,I lq ,P), then calculate the clear latent variables, and then pass the clear latent variables through the decoder to obtain the final clear image, completing the image restoration task.

[0046] The image restoration scheme based on text guidance and detail-preserving diffusion model in the embodiment of the present disclosure includes: step 1, obtaining a preliminary degraded image based on a clear image, and encoding the clear image into a latent space through an encoder and adding noise to obtain a noise latent variable; step 2, using the preliminary degraded image and the noise latent variable as conditions, inputting the conditional diffusion model to predict the noise added in the tth step, and then obtaining a latent variable that retains clear details through calculation, based on which a new degraded image is decoded through a decoder; step 3, re-inputting the new degraded image and the noise latent variable as conditions into the conditional diffusion model to obtain a final predicted noise, and calculating the loss based on the final predicted noise and the preliminary predicted noise to hybrid train the conditional diffusion model; step 4, saving the model weights of the trained conditional diffusion model, and using the DDIM inference process to complete the target image restoration process.

[0047] The beneficial effects of the embodiments of the present disclosure are:

[0048] 1. The method of the present invention improves a low-quality degraded image to automatically generate a high-quality image. This method avoids the limitations of manually designed degradation models and ensures the preservation of image details, thereby improving the overall quality of the restored image.

[0049] 2. This invention improves the learning objective of the diffusion model. During the training process, two different loss conditions are randomly selected to train the model, which improves the model's generalization ability to accept different degradation conditions while preserving image details to the greatest extent.

[0050] 3. The present invention supports users to precisely control the restoration of low-quality images through text conditions, ensuring the semantic consistency of the generated high-quality images. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0052] Figure 1 A flowchart of an image restoration method based on text guidance and detail-preserving diffusion model provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0053] The embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.

[0054] The following describes the embodiments of the present disclosure through specific examples, and those skilled in the art can easily understand other advantages and effects of the present disclosure from the contents disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all of the embodiments. The present disclosure can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present disclosure. It should be noted that, in the absence of conflict, the following embodiments and features in the embodiments can be combined with each other. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present disclosure.

[0055] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein can be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on this disclosure, it should be understood by those skilled in the art that an aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects described herein can be used to implement an apparatus and / or practice a method. In addition, other structures and / or functionalities other than one or more of the aspects described herein can be used to implement this apparatus and / or practice this method.

[0056] It should also be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present disclosure. The illustrations only show components related to the present disclosure and are not drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component can be changed at will, and the component layout type may also be more complicated.

[0057] Additionally, in the following description, specific details are provided to provide a thorough understanding of the examples. However, one skilled in the art will appreciate that the aspects described can be practiced without these specific details.

[0058] The embodiments of the present disclosure provide an image restoration method based on text guidance and detail-preserving diffusion model, which can be applied to image restoration processes in image processing scenarios.

[0059] See also Figure 1 , is a flow chart of an image restoration method based on text guidance and detail-preserving diffusion model provided by an embodiment of the present disclosure. Figure 1 As shown, the method mainly includes the following steps:

[0060] Step 1: Obtain a preliminary degraded image based on the clear image, and encode the clear image into the latent space through an encoder and add noise to obtain a noise latent variable;

[0061] In the specific implementation, a preliminary degraded image is obtained based on the high-quality image, the clear image is encoded into the latent space through the encoder, and noise is added to obtain the noise latent variable z t The specific process mainly includes:

[0062] Step 1.1, after obtaining the initial degraded image I lq When defining a degradation model, the high-quality image I hq Degenerate to get I lq , the degradation model is as follows:

[0063] I lq =φ ω (I hq )

[0064] Among them, φ ω (·) represents the existing artificial multiple degradation modeling operations, which design multiple factors and processes that may affect image quality, such as noise, blur, compression and other forms of distortion. The low-quality image obtained will show severe degradation to slight degradation, etc. lq Can and high quality image I hq Constitute training data pairs.

[0065] Step 1.2, after obtaining the noise latent variable z t When using the autoencoder in Stable Diffusion, the high-quality image I hq Mapped to the latent space variable z0, and then gradually add noise to z0. During the noise addition process, set the hyperparameters time step t and α t , with variance β t Gaussian noise of ∈(0,1) is added to the encoded latent variable z0 to generate the noisy latent variable z t , the noise addition formula at a certain time t is:

[0066] z0=E(I hq ),

[0067]

[0068] Where E(·) represents the autoencoder, t∈{1,…,T},∈~N(0,I), represents the noise of the standard Gaussian distribution, α t =1-βt,

[0069] Step 2: The initial degraded image and the noise latent variable are used as conditions and input into the conditional diffusion model to predict the noise added in step t. Then, the latent variable that retains clear details is calculated and decoded into a new degraded image through the decoder.

[0070] In the specific implementation, the initial degraded image and noise latent variable z t As a condition, it is sent to the conditional diffusion model to predict the noise added in step t Then the latent variables with clear details are obtained by calculation Then decode the new degraded image through the decoder The specific process is as follows:

[0071] Step 2.1: Initially degrade the image I lq and noise latent variable z t As a condition, it is sent to the conditional diffusion model to predict the noise added in step t

[0072]

[0073] Among them, ∈ θ (·) is a conditional diffusion model, and the required input is the latent variable z in the tth step t , denoising time step t, initial degraded image I lq As a condition, P is the text condition, and the conditional diffusion model result output is the predicted noise.

[0074] In this step, we use a conditional diffusion model that can accept both images and text as conditions, such as the ControlNet model. During the training phase, the text condition P is derived from the BLIP model to complete the image-to-text task:

[0075] P=BLIP(I hq ),

[0076] Where BLIP(·) represents the BLIP model to obtain a clear image I hq Due to the existence of the text condition P, the semantic consistency of the repaired image can be guaranteed, while satisfying the user's controllability of image repair.

[0077] Step 2.2: Use the noise ∈ predicted by the model θ (z t ,t,I lq ,P), the latent variable of the t-1 step is obtained by calculating the initial prediction and retaining the latent variables with clear details The calculation formula is as follows:

[0078]

[0079] Since this step is obtained by calculating This is not the result obtained by the denoising process of DDIM or DDPM. It is not the final clear result we want to get, but it is still partially degraded, but retains many details of the latent variables.

[0080] Step 2.3: Keep the latent variables with clear details Decode the new degraded image through the decoder

[0081]

[0082] Where D(·) is the decoder, which can retain the latent variables with clear details Decode to pixel space to get the new degraded image This is obtained It not only retains some degradation, but also retains many image details. What is important is that the new degraded image It can be automatically generated by the diffusion model at different time steps t, without the need to manually design a complex degradation process.

[0083] Step 3: Re-input the new degraded image and noise latent variables as conditions into the conditional diffusion model to obtain the final predicted noise. The loss is calculated based on the final predicted noise and the preliminary predicted noise to hybrid train the conditional diffusion model.

[0084] In specific implementation, according to the new degraded image and noise latent variable z t The noise is then fed back into the conditional diffusion model as a condition to obtain the final predicted noise. The specific process of hybrid training the diffusion model by calculating the loss using the final predicted noise and the preliminary predicted noise may include:

[0085] Step 3.1, based on the new degraded image and noise latent variable z t As a condition, it is re-entered into the conditional diffusion model to obtain the final predicted noise ∈ t .

[0086] The input condition of the model in this process is the original initial degraded image I lq Replace the degraded image with a new one Using the explicitly constrained diffusion iterative process, the noise latent variable at step t-1 can be predicted by the following operations:

[0087]

[0088] The noise latent variable z obtained through this process t-1 More retained from I lq The image details are clearly shown, which effectively solves the instability problem of the diffusion model.

[0089] In step 3.2, the diffusion model is trained by mixing the final predicted noise and the preliminary predicted noise to calculate the loss.

[0090] In order to make the model training more generalizable, we use hybrid training to enable the model to be able to lq , new degraded image Even more image degradation types maintain good fitting ability, setting a loss to calculate the probability of two losses respectively when p iide and 1-p iide , during training, with p iide The probability loss is defined as the prediction noise The mean square error between ∈ and the actual noise:

[0091]

[0092] With 1-p iide The probability loss is defined as the prediction noise ∈ θ (z t ,t,I lq ,P) and the mean square error between the actual noise ∈:

[0093]

[0094] where p iide ∈(0,1), where U(1,T) represents a uniform distribution on the set 1,…,T, and N(0,I) represents a multidimensional standard Gaussian distribution with a mean of 0 and a covariance matrix of the unit matrix I. These two losses are used to constrain and z t-1 Maintain the similarity to the greatest extent possible to ensure that the final predicted z t-1 Be able to retain as much as possible from I lq image details.

[0095] Step 4: Save the model weights of the trained conditional diffusion model and use the DDIM reasoning process to complete the target image restoration process.

[0096] In the specific implementation, we can first sample a noise latent variable z from N(0,I) t and the corresponding low-quality image I lq , for different time steps sampled by DDIM, the trained conditional diffusion model ∈θ The model weights are used to predict the noise ∈ θ (z t ,t,I lq ,P), perform step-by-step denoising, and then predict a clear latent variable z0, and then pass z0 through the decoder to obtain the final clear image to complete the image restoration task.

[0097] The image restoration method based on text guidance and detail-preserving diffusion model provided in this embodiment uses the generation prior of the diffusion model for image restoration technology to design a method that can guide it to generate high-quality images based on given low-quality images during the generation process, ensuring that the restored high-quality image is faithful to the original content. Compared with the method that only uses low-quality images obtained by artificial degradation modeling as training conditions, our method can successfully generate high-fidelity restoration images with vivid colors and highly realistic details when low-quality input images have multiple unknown degradations. In addition, our method supports users to precisely control the restoration of images through text conditions, ensuring semantic consistency. Through the method disclosed in this disclosure, the model's generalization ability to accept different degradation conditions is improved, image details are preserved to the greatest extent, and users are supported to precisely control the restoration of low-quality images through text conditions.

[0098] It should be understood that various parts of the present disclosure can be implemented in hardware, software, firmware, or a combination thereof.

[0099] The above description is merely a specific embodiment of the present disclosure, but the scope of protection of the present disclosure is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this disclosure should be included in the scope of protection of the present disclosure. Therefore, the scope of protection of the present disclosure should be based on the scope of protection of the claims.

Claims

1. An image restoration method based on text guidance and detail-preserving diffusion model, characterized in that: include: Step 1: Obtain a preliminary degraded image based on the clear image, and encode the clear image into the latent space through an encoder and add noise to obtain a noise latent variable; Step 2: The initial degraded image and the noise latent variable are used as conditions and input into the conditional diffusion model to predict the noise added in step t. Then, the latent variable that retains clear details is calculated and decoded into a new degraded image through the decoder. Step 3: Re-input the new degraded image and noise latent variables as conditions into the conditional diffusion model to obtain the final predicted noise. The loss is calculated based on the final predicted noise and the preliminary predicted noise to hybrid train the conditional diffusion model. Step 4: Save the model weights of the trained conditional diffusion model and use the DDIM reasoning process to complete the target image restoration process.

2. The method according to claim 1, characterized in that The step 1 specifically includes: Step 1.1, define a degradation model to transform the clear image I hq Degrade to obtain the initial degraded image I lq , where the expression of the degradation modeling is I lq =φ ω (I hq ) Among them, φ ω (·) represents various preset degradation modeling operations; Step 1.2, use the preset autoencoder to transform the clear image I hq Mapped to the latent space variable z0, and then gradually noised the latent space variable z0. During the noise addition process, the hyperparameters time step t and α are set. t , with variance β t Gaussian noise of ∈(0,1) is added to the encoded latent space variable z0 to generate the noisy latent variable z t , where the expression of the latent space variable is the noise addition formula at time t is z0=E(I hq ); The noise addition formula at time t is: Where E(·) represents the autoencoder, t∈{1,…,T},∈~N(0,I), represents the noise of the standard Gaussian distribution, 3. The method according to claim 2, characterized in that The step 2 specifically includes: Step 2.1, the initial degraded image I lq and noise latent variable z t As a condition, it is sent to the conditional diffusion model to predict the noise added in step t Among them, ∈ θ (·) is the conditional diffusion model, z t is the noise latent variable of the tth step, t is the denoising time step, and the initial degraded image I lq As a condition, P is the text condition corresponding to the clear image; Step 2.2, using the noise ∈ predicted by the model θ (z t ,t,I lq ,P), and calculate the hidden variables of the t-1 step to get the preliminary prediction and latent variables that retain clear details Step 2.3, the latent variables with clear details will be retained Decode the new degraded image through the decoder in, is the decoder, which is used to retain the latent variables with clear details Decode to pixel space to get the new degraded image 4. The method according to claim 3, characterized in that Before step 2.1, the method further includes: Use the BLIP model to obtain the text conditions corresponding to the clear image P=BLIP(I hq ) Here, BLIP(·) represents the BLIP model.

5. The method according to claim 4, characterized in that The step 3 specifically includes: Step 3.1, based on the new degraded image and noise latent variable z t As a condition, it is re-sent into the conditional diffusion model to obtain the final predicted noise ∈ t ; Step 3.2, using the noise ∈ predicted by the model t , the final predicted hidden variable z of step t-1 is obtained by calculation t-1 Step 3.3, calculate the final predicted noise ∈ t The mean square error between the actual noise ∈ is used as the first loss Among them, E represents the mathematical expectation of different sampling variables, that is, taking the average value in multiple samples, U(1,T) represents the uniform distribution on the set 1,…,T, N(0,I) represents the multidimensional standard Gaussian distribution with mean 0 and covariance matrix as the unit matrix I, Represents the square of the L2 norm, that is, the square of the Euclidean distance, which is used to measure the gap between the predicted noise and the actual noise; Step 3.4, calculate the noise of the initial prediction The mean square error between the actual noise ∈ is used as the second loss In step 3.5, the conditional diffusion model is trained jointly based on the first loss and the second loss.

6. The method according to claim 5, characterized in that The step 4 specifically includes: Noise from a standard Gaussian distribution A noise latent variable z is sampled from t and the corresponding preliminary degraded image I lq , for different time steps sampled by DDIM, the trained conditional diffusion model ∈ θ The model weights are used to predict the noise ∈ at each time step t θ (z t ,t,I lq ,P), then calculate the clear latent variables, and then pass the clear latent variables through the decoder to obtain the final clear image, completing the image restoration task.

Images