Image inversion method based on diffusion probability model
By optimizing text embedding vectors to achieve high-precision real image reconstruction, the problem that the denoising diffusion probability model is difficult to accurately invert the real image, and the precise reconstruction and editing of the real image is achieved.
Patent Information
- Application Number
- CN202411773284.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-04
- Publication Date
- 2025-03-11
AI Technical Summary
The existing denoising and diffusion probability model is difficult to accurately invert real images, mainly because the distribution of the real images and the generated images are inconsistent, resulting in unstable reconstruction quality.
By treating the text embedding vector as an optimizeable parameter and optimizing based on the reconstruction loss function, combining the deterministic noise addition and denoising algorithm of the denoising diffusion implicit model, high-precision real image reconstruction is achieved.
Accurate reconstruction of real images and support image editing, which significantly improves the ability of denoising diffusion probability model to invert and edit real images.
Smart Images
Figure CN119600143A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image inversion, and more specifically, to an image inversion method based on a diffusion probability model. Background Art
[0002] The goal of real image inversion is to find a latent vector that, when forward propagated through the generative model, can reconstruct or approximate the representation of the original input image. By inverting the real image into the latent space, it should be possible to perform various edits and operations on the image without significantly degrading the image quality.
[0003] Although DDPM (denoising diffusion probability model) has achieved amazing results in unconditional image generation and text-controlled image generation, there is a significant gap between the distribution of real-world images and generated images, which makes it impossible to accurately reconstruct real images based on DDPM. An intuitive image inversion method based on DDPM is to add a certain amount of Gaussian noise to the real image, and then reversely reconstruct the original image by performing a predefined number of diffusion steps. However, this method often cannot accurately reconstruct the original image due to the introduction of random Gaussian noise in the denoising process, resulting in unstable reconstruction quality. The current mainstream is a deterministic image inversion method based on DDIM (denoising diffusion implicit model). This method avoids the introduction of randomness through a deterministic denoising and denoising method, greatly improving the success rate of image inversion, but this deterministic denoising algorithm can also lead to reconstruction failures. The latent vector obtained by the DDIM-based inversion method is also a Gaussian distribution, but it has a larger variance than the standard Gaussian distribution. Since the DDPM model only learns to generate images from the standard Gaussian distribution during the training phase, it is difficult to generalize to such a distribution, which shows that the complexity of the real-world domain is far beyond the scope of what the model can directly learn. Summary of the invention
[0004] In order to overcome the above-mentioned defects of the prior art, the present invention provides an image inversion method based on a diffusion probability model, which regards the text embedding vector as a parameter that can be optimized and optimizes it based on a reconstruction loss function to achieve high-precision real image reconstruction; by combining the image inversion method with an image editing method based on a text-controlled image generation model, real image editing can be accurately achieved.
[0005] To achieve the above object, the present invention provides the following technical solutions:
[0006] An image inversion method based on a diffusion probability model comprises the following steps:
[0007] Step S1, a deterministic denoising algorithm based on a denoising diffusion implicit model is used to perform multi-step denoising on the input real image to generate a latent vector;
[0008] Step S2, generating a text embedding vector according to the corresponding text description of the input image, and using the text embedding vector as an optimizable parameter;
[0009] Step S3, inputting the generated latent vector and the text embedding vector into the stable diffusion model, and performing a denoising process through the denoising algorithm of the denoising diffusion implicit model to generate a reconstructed image;
[0010] Step S4, calculating and using the root mean square error between the reconstructed image and the real image as the optimization loss function, and optimizing the text embedding vector by a gradient descent algorithm;
[0011] Step S5, output the final reconstructed image and complete the real image inversion.
[0012] As a further solution of the present invention, in step S4, based on the Markov process of the denoising diffusion probability model, the acceleration operation is performed by ensuring that the noisy picture generated by the model at each time step is consistent with the noisy picture inverted based on the denoising diffusion implicit model, and the difference between the noisy latent code generated by the model at each time step and the reference noisy latent code obtained by inverting the denoising diffusion implicit model is calculated.
[0013] As a further solution of the present invention, in step S4, the stable diffusion model performs noise addition and denoising operations in the latent space, and optimizes the text embedding vector based on the loss function of accelerated training, and the formula of the loss function is:
[0014]
[0015] Where L(θ) is the loss function, θ is the text embedding vector parameter, is the noisy latent code inverted by the denoised diffusion implicit model at time t-1, z t-1 is the noisy latent code predicted by the stable diffusion model at time t-1, θ t is the text embedding vector parameter at time step t, As auxiliary conditions, is the input latent code at time step t.
[0016] As a further solution of the present invention, step S4 further includes: maintaining a corresponding text embedding vector parameter for each time step A set of parameters for all time steps is constructed, with the optimization goal of minimizing the error between the reference latent code and the model predicted latent code at each time step. The text embedding vector parameters of each time step are independently optimized based on the global loss function.
[0017] As a further solution of the present invention, the formula of the global loss function is:
[0018]
[0019] Where L(θ) is the global loss function, θ = {θ1, θ2, ..., θ T} is a collection of all time step parameters, is the noisy latent code inverted by the denoised diffusion implicit model at time t-1, z t-1 is the noisy latent code predicted by the stable diffusion model at time t-1, θ t is the text embedding vector parameter at time step t, As auxiliary conditions, is the input latent code at time step t, and T is the total number of time steps.
[0020] Compared with the prior art, the image inversion method based on the diffusion probability model of the present invention has the following beneficial effects:
[0021] The present invention discusses the problem that the denoising diffusion probability model is difficult to invert the real image, and analyzes that the main reason is the distribution inconsistency between the real image and the generated image. In order to solve this problem, for the text-controlled image generation diffusion probability model, an image inversion method based on text embedding vector optimization is proposed. This method regards the text embedding vector of each time step in the denoising process as a learnable parameter to reconstruct the original image to support the editing of the real image. This method can achieve accurate image reconstruction and can be combined with the image editing method to edit the real image, effectively improving the ability of the denoising diffusion probability model to invert and edit the real image. BRIEF DESCRIPTION OF THE DRAWINGS
[0022] Figure 1 The figure is a flow chart of an image inversion method based on a diffusion probability model according to the present invention.
[0023] Figure 2 Flowchart of the image inversion algorithm optimized based on text embedding vectors.
[0024] In the figure, text prompt: text description; text embedding: text embedding vector; Trainable parameters: trainable parameters; DDIM Inversion: deterministic diffusion implicit model inversion; x ori : real image; x T : Hidden vector; Stable Diffusion: Stable diffusion model; Frozen: Frozen; x rec : Reconstructed image; MSE loss: mean square error loss DETAILED DESCRIPTION
[0025] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0026] Example 1
[0027] An image inversion method based on a diffusion probability model comprises the following steps:
[0028] Step S1, a deterministic denoising algorithm based on a denoising diffusion implicit model is used to perform multi-step denoising on the input real image to generate a latent vector;
[0029] Step S2, generating a text embedding vector according to the corresponding text description of the input image, and using the text embedding vector as an optimizable parameter;
[0030] Step S3, inputting the generated latent vector and the text embedding vector into the stable diffusion model, and performing a denoising process through the denoising algorithm of the denoising diffusion implicit model to generate a reconstructed image;
[0031] Step S4, calculating and using the root mean square error between the reconstructed image and the real image as the optimization loss function, optimizing the text embedding vector by a gradient descent algorithm, so that the generated reconstructed image can accurately restore the real image;
[0032] Step S5, output the final reconstructed image and complete the real image inversion.
[0033] Image inversion includes two goals: first, to reconstruct the real-world image x based on the model ori Get the reconstructed image x rec The latent vector corresponding to the image, where x ori and x rec It should be as close as possible. Secondly, by manipulating the latent vector, the image can be edited and reconstructed, including style transfer, age changes, etc.
[0034] To address the problem of inconsistent distribution between the generated image domain and the real image domain, the embodiment of the present invention adopts a deterministic denoising diffusion hidden model instead of a denoising diffusion probability model. Contrary to the image generation task, the image inversion performs a diffusion process in the opposite direction, where x0 is set to a given image.
[0035] In step S1 of the embodiment of the present invention, the latent vector of the real image is generated by the following noise addition formula:
[0036]
[0037] In the formula, x t is the noisy image at time t, x t+1 is the noisy image at time t+1, α t+1 is the diffusion coefficient preset by the model at time t+1, α t is the diffusion coefficient preset by the model at time t, is the cumulative diffusion coefficient from the starting time step to the current time step t during the diffusion process, ∈ θ (x t , t) is the output of the denoising diffusion implicit model.
[0038] For text-controlled image generation, although the DDIM-based deterministic denoising and denoising method cannot completely reconstruct the real image, it can still serve as a promising starting point for subsequent optimization, so that it can efficiently achieve high-precision reconstruction of the real image.
[0039] In the embodiment of the present invention, a noise addition formula is used to generate a latent vector, and the text embedding vector information is used as a learnable parameter and the root mean square error is used as an optimized loss function to learn the text embedding vector information.
[0040] In the traditional denoising diffusion probability model, there is usually a significant difference between the generated image and the real input image, making it difficult to achieve high-precision inversion and reconstruction of the real image. In order to evaluate the reconstruction quality of the generated image and the optimization effect of the model, the mean square error in the pixel space is often used as the loss function. The function of this formula is to minimize the pixel error between the reconstructed image and the real image, guide the model to adjust the optimization parameters, so that the generated image is as close to the input image as possible, and ensure that the model has basic inversion capabilities and reconstruction accuracy. The formula is defined as follows:
[0041]
[0042] In the formula, x rec is the image generated by the corresponding model, x ori is the corresponding real image, is the square of the two-norm, which represents the Euclidean distance in pixel space.
[0043] Considering that the entire denoising process of the denoising diffusion implicit model requires multiple iterations, although the optimization target has a small number of parameters, it still brings huge resource consumption. The Markov process based on the denoising diffusion probability can accelerate the operation by ensuring that the noisy image generated by the model at each time step t is consistent with the image inverted and noisy based on the denoising diffusion implicit model.
[0044] In step S4 in the embodiment of the present invention, based on the Markov process of the denoising diffusion probability model, the acceleration operation is performed by ensuring that the noisy picture generated by the model at each time step is consistent with the noisy picture inverted based on the denoising diffusion implicit model, and the difference between the noisy latent code generated by the model at each time step and the reference noisy latent code obtained by inverting the denoising diffusion implicit model is calculated.
[0045] In step S4 in the embodiment of the present invention, the stable diffusion model performs noise addition and denoising operations in the latent space, and optimizes the text embedding vector based on the loss function of accelerated training, and the formula of the loss function is:
[0046]
[0047] Where L(θ) is the loss function, θ is the text embedding vector parameter, is the noisy latent code inverted by the denoised diffusion implicit model at time t-1, z t-1 is the noisy latent code predicted by the stable diffusion model at time t-1, θ t is the text embedding vector parameter at time step t, As auxiliary conditions, is the input latent code at time step t.
[0048] In order to achieve higher accuracy reconstruction, a corresponding text embedding vector parameter is maintained for each time step t
[0049] The step S4 in the embodiment of the present invention further comprises: maintaining a corresponding text embedding vector parameter for each time step A set of parameters for all time steps is constructed, with the optimization goal of minimizing the error between the reference latent code and the model predicted latent code at each time step. The text embedding vector parameters of each time step are independently optimized based on the global loss function.
[0050] The formula of the global loss function in the embodiment of the present invention is:
[0051]
[0052] Where L(θ) is the global loss function, θ = {θ1, θ2, ..., θ T} is a collection of all time step parameters, is the noisy latent code inverted by the denoised diffusion implicit model at time t-1, z t-1 is the noisy latent code predicted by the stable diffusion model at time t-1, θ t is the text embedding vector parameter at time step t, As auxiliary conditions, is the input latent code at time step t, and T is the total number of time steps.
[0053] This form allows each θ t Optimizes independently of other time steps, potentially better suited to the data characteristics and needs of each specific time step.
[0054] Example 2
[0055] like Figure 2 The flowchart of the image inversion algorithm based on text embedding vector optimization of the present invention is shown. First, starting from the input text description, a text embedding vector is generated and used as a trainable parameter for optimization; after the real image is input, the corresponding latent vector is generated using the deterministic diffusion implicit model inversion algorithm; then, the generated text embedding vector and the latent vector are input into the stable diffusion model together, and the model processes the input data in a frozen state and generates a reconstructed image through a denoising diffusion process; finally, the mean square error between the reconstructed image and the real image is used as the optimization target, and the text embedding vector parameters are adjusted to optimize the model so that the generated reconstructed image is closer to the real image; the whole process integrates the steps of text description, latent vector generation, stable diffusion model calculation and error feedback optimization to achieve high-precision image inversion.
[0056] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.
[0057] Finally: The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.
Claims
1. An image inversion method based on a diffusion probability model, characterized in that: The following steps are involved: Step S1, a deterministic denoising algorithm based on a denoising diffusion implicit model is used to perform multi-step denoising on the input real image to generate a latent vector; Step S2, generating a text embedding vector according to the corresponding text description of the input image, and using the text embedding vector as an optimizable parameter; Step S3, inputting the generated latent vector and the text embedding vector into the stable diffusion model, and performing a denoising process through the denoising algorithm of the denoising diffusion implicit model to generate a reconstructed image; Step S4, calculating and using the root mean square error between the reconstructed image and the real image as the optimization loss function, and optimizing the text embedding vector by a gradient descent algorithm; Step S5, output the final reconstructed image and complete the real image inversion.
2. The image inversion method based on the diffusion probability model according to claim 1, characterized in that: In step S4, based on the Markov process of the denoising diffusion probability model, the acceleration operation is performed by ensuring that the noisy picture generated by the model at each time step is consistent with the noisy picture inverted based on the denoising diffusion implicit model, and the difference between the noisy latent code generated by the model at each time step and the reference noisy latent code inverted by the denoising diffusion implicit model is calculated.
3. The image inversion method based on the diffusion probability model according to claim 1 or 2, characterized in that: In step S4, the stable diffusion model performs noise addition and denoising operations in the latent space, and optimizes the text embedding vector based on the loss function of accelerated training. The formula of the loss function is: Where L(θ) is the loss function, θ is the text embedding vector parameter, is the noisy latent code inverted by the denoised diffusion implicit model at time t-1, z t-1 is the noisy latent code predicted by the stable diffusion model at time t-1, θ t is the text embedding vector parameter at time step t, As auxiliary conditions, is the input latent code at time step t.
4. The image inversion method based on the diffusion probability model according to claim 1, characterized in that: The step S4 further comprises: maintaining a corresponding text embedding vector parameter for each time step A set of parameters for all time steps is constructed, with the optimization goal of minimizing the error between the reference latent code and the model predicted latent code at each time step. The text embedding vector parameters of each time step are independently optimized based on the global loss function.
5. The image inversion method based on the diffusion probability model according to claim 1 or 4, characterized in that: The formula of the global loss function is: Where L(θ) is the global loss function, θ={θ1,θ2,…,θ T } is a collection of all time step parameters, is the noisy latent code inverted by the denoised diffusion implicit model at time t-1, z t-1 is the noisy latent code predicted by the stable diffusion model at time t-1, θ t is the text embedding vector parameter at time step t, C is the auxiliary condition, is the input latent code at time step t, and T is the total number of time steps.