Image inversion method and device based on diffusion model, terminal and storage medium

Abstract

The application discloses an image inversion method and device based on a diffusion model, a terminal and a storage medium. The method comprises the following steps: inputting an initial image into an encoder to determine an initial latent vector corresponding to the initial image; obtaining a text prompt; and using a diffusion inversion model to perform inversion and denoising on the initial latent vector based on the text prompt to determine a target latent vector. The diffusion inversion model is implemented by modeling noise in the inversion and denoising process of a denoising diffusion implicit model using a latent bias vector. The denoising diffusion implicit model is implemented based on a UNet model. The target latent vector is decoded using a decoder to determine a target image. The application models the bias between two adjacent latent vectors by using a latent bias vector, can obtain an optimal initial noise without introducing additional parameters, and effectively solves the problem that existing inversion methods based on the diffusion model cannot guarantee high-quality and high-accuracy reconstruction of an image while achieving low computational overhead.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image processing, and in particular, to an image inversion method and device based on a diffusion model, a terminal and a storage medium. BACKGROUND

[0002] The inversion of real images originally originated from GAN (Generative Adversarial Network) for image editing. Existing image editing methods usually edit the latent code corresponding to the image in the latent space to change the generated image. Since StyleGAN (Style-based Generative Adversarial Network) has achieved the most advanced image quality and rich latent space semantics in GAN image generation, many inversion methods use StyleGAN to complete this task. With the development of diffusion models, they have surpassed GAN in diversity and fidelity of generation, so inversion based on diffusion models has gradually become a hot topic.

[0003] The inversion of diffusion models aims to find a combination of appropriate latent noise and conditional prompt embedding for a given image, which is a key step for diffusion models to operate on real images. Existing diffusion model inversion techniques are mainly divided into two categories: numerical iteration and optimization-based. Although these methods have made some improvements, they either introduce additional cumbersome parameters, increasing the computational burden, or have poor stability due to the limitations of numerical iteration.

[0004] Therefore, the existing inversion methods based on diffusion models cannot guarantee high-quality and high-accuracy reconstruction of images while achieving low computational overhead.

[0005] Therefore, the existing technology needs to be improved and developed. SUMMARY

[0006] The technical problem to be solved by the present application is to provide an image inversion method and device based on a diffusion model, a terminal and a storage medium, which can solve the problem that existing inversion methods based on diffusion models cannot guarantee high-quality and high-accuracy reconstruction of images while achieving low computational overhead.

[0007] The technical solution adopted by the present application to solve the problem is as follows:

[0008] In a first aspect, the present application provides an image inversion method based on a diffusion model, wherein the method comprises:

[0009] inputting an initial image into an encoder to determine an initial latent vector corresponding to the initial image;

[0010] obtaining a text prompt, and performing inversion and denoising on the initial latent vector based on the text prompt by using a diffusion inversion model to determine a target latent vector, wherein the diffusion inversion model is implemented by modeling noise in a denoising diffusion implicit model inversion and denoising process by using a latent bias vector, and the denoising diffusion implicit model is implemented based on a UNet model;

[0011] decoding the target latent vector by using a decoder to determine a target image.

[0012] In an implementation method, the step of performing inversion and denoising on the initial latent vector based on the text prompt by using the diffusion inversion model to determine the target latent vector comprises:

[0013] constructing a first optimization function corresponding to the diffusion inversion model, and performing inversion and denoising on the initial latent vector based on the text prompt and the first optimization function by using the diffusion inversion model to optimize the latent bias vector;

[0014] when the latent bias vector is optimal, determining that a latent vector output by the diffusion inversion model is the target latent vector.

[0015] In an implementation method, the first optimization function comprises:

[0016]

[0017] wherein t is a time step, b t is the latent bias vector in the inversion process, is the latent bias vector in the denoising process, α t is a parameter predefined in the diffusion inversion model, C is a text prompt, and ∈ θ is the pre-trained UNet model, z t is a latent vector after inversion of the initial latent vector for t time steps.

[0018] In an implementation method, the method for determining the target latent vector further comprises:

[0019] constructing a second optimization function, and performing inversion and denoising on the initial latent vector based on an empty text prompt, the text prompt, and the second optimization function by using the diffusion inversion model to optimize the latent bias vector;

[0020] when the latent bias vector is optimal, determining that a latent vector output by the diffusion inversion model is the target latent vector.

[0021] In an implementation method, the second optimization function comprises:

[0022]

[0023] where t is a time step, is the latent bias vector in the inversion process, is the latent bias vector in the denoising process,

[0024]

[0025]

[0026]

[0027] where, α t is predefined by the parameters in the diffusion inversion model, is the empty text prompt, C is the text prompt, is the pre-trained UNet model, λ>1, z t is the latent vector after t time steps of inversion of the initial latent vector.

[0028] In an implementation method, before the initial latent vector is inverted and denoised, further comprising:

[0029] Parameter freezing is performed on the encoder and the decoder;

[0030] A loss function is constructed based on peak signal-to-noise ratio, structural similarity, and perceptual similarity;

[0031] The initial latent vector is optimized using the parameter-frozen encoder, the decoder, and the loss function.

[0032] In an implementation method, the initial latent vector is optimized using the parameter-frozen encoder, the decoder, and the loss function, comprising:

[0033] Initialization represents the optimized initial latent vector, x0 is the initial image, z0 represents the initial latent vector, and E is the encoder;

[0034] is optimized:

[0035]

[0036] where, is the decoder, is the loss function; ​

[0037]

[0038] wherein, is a peak signal-to-noise ratio loss, is a structural similarity loss, is a perceptual similarity loss.

[0039] In a second aspect, the embodiments of the present application further provide an image inversion device based on a diffusion model, wherein the image inversion device based on the diffusion model comprises:

[0040] An image encoding module is configured to input an initial image into an encoder to determine an initial latent vector corresponding to the initial image.

[0041] An inversion and denoising module is configured to obtain a text prompt, and perform inversion and denoising on the initial latent vector based on the text prompt by using a diffusion inversion model, to determine a target latent vector, wherein the diffusion inversion model is achieved by modeling noise in an inversion and denoising process of a denoising diffusion implicit model by using a latent bias vector, and the denoising diffusion implicit model is achieved based on a UNet model.

[0042] An image decoding module is configured to decode the target latent vector by using a decoder to determine a target image.

[0043] In a third aspect, the embodiments of the present application further provide a terminal, which comprises a memory and one or more processors; the memory stores one or more programs; the programs contain instructions for executing the image inversion method based on the diffusion model as described above; and the processor is configured to execute the programs.

[0044] In a fourth aspect, the embodiments of the present application further provide a computer readable storage medium, which stores a plurality of instructions, wherein the instructions are adapted to be loaded and executed by a processor to implement the image inversion method based on the diffusion model as described above.

[0045] The beneficial effects of the present application: the embodiment of the present application inputs an initial image into an encoder, determines an initial latent vector corresponding to the initial image, obtains a text prompt, and uses a diffusion inversion model to invert and denoise the initial latent vector based on the text prompt, to determine a target latent vector, wherein the diffusion inversion model is realized by modeling the noise in the denoising diffusion implicit model inversion and denoising process using a latent bias vector, and the denoising diffusion implicit model is realized based on a UNet model; the target latent vector is decoded using a decoder to determine a target image. The present application models the bias between two adjacent latent vectors by using a latent bias vector, which can obtain an optimal initial noise without introducing additional parameters, and effectively solves the problem that existing inversion methods based on diffusion models cannot guarantee high-quality and high-accuracy reconstruction of images while achieving low computational overhead. BRIEF DESCRIPTION OF DRAWINGS

[0046] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments described in the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0047] Figure 1 is a flowchart of the image inversion method based on the diffusion model provided by the embodiment of the present application.

[0048] Figure 2 is a latent bias vector optimization process diagram provided by the embodiment of the present application.

[0049] Figure 3 is an image information projection diagram provided by the embodiment of the present application.

[0050] Figure 4 is an internal module diagram of the image inversion device based on the diffusion model provided by the embodiment of the present application.

[0051] Figure 5 is a principle block diagram of the terminal provided by the embodiment of the present application. DETAILED DESCRIPTION

[0052] The present application discloses an image inversion method, device, terminal and storage medium based on a diffusion model. In order to make the purpose, technical scheme and effect of the present application more clear and explicit, the present application will be further described in detail with reference to the drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and not to limit the present application.

[0053] It will be further understood that the terms "comprises" and / or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. As used herein, "connected," "coupled," and "coupling" can include wirelessly connected, wirelessly coupled, or wireless coupling. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0054] Those skilled in the art can understand that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. It should also be understood that terms such as those defined in a general dictionary should be understood to have meanings consistent with meanings in the context of the prior art, and should not be interpreted to have idealized or overly formal meanings unless specifically defined as such herein.

[0055] The inversion of real images originally originated from GAN (Generative Adversarial Network) for image editing. Existing image editing methods usually edit the latent code corresponding to the image in the latent space to achieve the effect of changing the generated image. Since StyleGAN has achieved the most advanced image quality and rich latent space semantics in image generation of GAN, many inversion methods use StyleGAN to complete this task. With the development of diffusion models, it has surpassed GAN in diversity and fidelity of generation, so inversion based on diffusion models has gradually become a hot spot.

[0056] The inversion of diffusion models aims to find a combination of appropriate latent noise and conditional prompt embedding for a given image, which is a key step for diffusion models to operate on real images. Existing diffusion model inversion techniques are mainly divided into two categories: numerical iteration and optimization-based. Although these methods have made some improvements, they either introduce additional cumbersome parameters, increasing the computational burden, or have poor stability due to the limitations of numerical iteration.

[0057] Therefore, existing diffusion model-based inversion methods cannot guarantee high-quality and high-accuracy reconstruction of images while achieving low computational overhead.

[0058] In view of the above defects of the prior art, the present application provides an image inversion method based on a diffusion model, which comprises the following steps: inputting an initial image into an encoder to determine an initial latent vector corresponding to the initial image; obtaining a text prompt, and using a diffusion inversion model to perform inversion and denoising on the initial latent vector based on the text prompt to determine a target latent vector, wherein the diffusion inversion model is implemented by using a latent bias vector to model noise in the denoising diffusion implicit model inversion and denoising process, and the denoising diffusion implicit model is implemented based on a UNet model; and using a decoder to decode the target latent vector to determine a target image. The present application models the bias between two adjacent latent vectors by using a latent bias vector, can obtain an optimal initial noise without introducing additional parameters, and effectively solves the problem that existing inversion methods based on the diffusion model cannot guarantee high-quality and high-accuracy reconstruction of an image while achieving low computational overhead.

[0059] Exemplary method

[0060] As shown in Figure 1 , the method comprises the following steps:

[0061] Step S100: inputting an initial image into an encoder to determine an initial latent vector corresponding to the initial image.

[0062] As shown in Figure 2 , the high-dimensional initial image is encoded into a low-dimensional initial latent vector z0 in the latent space by the encoder, so as to obtain an optimal initial noise by inversion based on the initial latent vector, thereby reconstructing a high-quality target image. The encoder in the present embodiment is a VQ-VAE (vector quantization variational autoencoder), which is implemented by an encoder, a quantizer, a decoder and a loss function. The input data is encoded into a latent representation by the encoder, and the discrete representation from the quantizer is accepted by the decoder to generate data matching the original input. The vector quantization variational autoencoder combines the variational autoencoder and the quantization method, and can generate high-quality data.

[0063] As shown in Figure 1 , the method further comprises the following steps:

[0064] Step S200: obtaining a text prompt, and using a diffusion inversion model to perform inversion and denoising on the initial latent vector based on the text prompt to determine a target latent vector, as shown in Figure 2 . , wherein the diffusion inversion model is implemented by using a latent bias vector to model noise in the denoising diffusion implicit model inversion and denoising process, and the denoising diffusion implicit model is implemented based on a UNet model.

[0065] The conditional text prompt obtained by the CLIP (Contrastive Language-Image Pre-training) text encoder or the text prompt provided by the user.

[0066] The formula of the DDIM (Denoising Diffusion Implicit Models) when performing denoising is as follows:

[0067]

[0068] where t is the time step, z t is the latent vector generated at time step t, a t is a parameter predefined by the DDIM scheduler, C is the embedding of the conditional text prompt obtained by the pre-trained CLIP text encoder, and ∈ θ represents the pre-trained UNet model.

[0069] According to the above formula, the formula of the inversion process can be expressed as:

[0070]

[0071] Based on the above formula, z t is calculated at time t, and since z t depends on ∈ θ (z t , t, C), and ∈ θ (z t , t, C) in turn depends on z t , a circular dependency is formed. One solution is to approximate ∈ θ (z t , t, C) ≈ ∈ θ (z t-1 , t, C), which introduces an error in each inversion step due to this approximation assumption. This method is difficult to produce accurate reconstruction when encountering images with rich details or large smooth regions.

[0072] In view of the above problems, the present embodiment introduces a latent bias vector into the DDIM to model the latent bias between adjacent inversion steps through the latent bias vector to find the optimal inversion trajectory. In the present embodiment, a latent bias vector of the same dimension as the latent vector in the LDMs (Latent Diffusion Models) is introduced to define the forward (inversion) and backward (denoising) transmission. The mathematical representation of the forward (inversion) and backward (denoising) transmission defined by the latent bias vector is as follows:

[0073] z t = z t-1 + bt ,

[0074]

[0075] where t is a time step, z t is the latent vector generated by the initial latent vector at time step t, b t is the latent bias vector in the inversion process, is the latent bias vector in the denoising process, by z t is input into the UNet prediction calculation.

[0076] In an implementation manner, the adopting the diffusion inversion model to invert and denoise the initial latent vector based on the text prompt to determine the target latent vector comprises:

[0077] constructing a first optimization function corresponding to the diffusion inversion model, and adopting the diffusion inversion model to invert and denoise the initial latent vector based on the text prompt and the first optimization function to optimize the latent bias vector;

[0078] when the latent bias vector is optimal, determining that the latent vector output by the diffusion inversion model is the target latent vector.

[0079] In short, the embodiment constructs a first optimization function corresponding to the diffusion inversion model, and optimizes the latent bias vector according to the first optimization function in the process of denoising and inverting the initial latent vector based on the text prompt by the diffusion inversion model, so as to obtain the optimal latent bias vector, that is, the optimal initial noise, so that the target latent vector obtained by the diffusion inversion model based on the optimal latent bias vector and the text prompt to invert and denoise the initial latent vector can reconstruct an image with better accuracy. In addition, the embodiment can effectively map the noise sample most matched with the original image from the latent space by optimizing the latent bias vector, thereby providing a new way for image editing, theme-driven image generation and other downstream tasks.

[0080] In an implementation manner, the latent bias vector is optimized by the first optimization function to align the inversion and denoising (generation) trajectories, wherein the first optimization function comprises:

[0081]

[0082] According to the denoising diffusion implicit model scheduler formula:

[0083]

[0084] The formula of the latent bias vector is:

[0085]

[0086] wherein, α t is a predefined parameter in the diffusion inversion model or the denoising diffusion implicit model, C is the text prompt, ∈ θ is a pre-trained UNet model.

[0087] In an implementation, the denoising diffusion implicit model is applied to z t provides a reasonable estimate, the latent bias vector can be initialized as:

[0088]

[0089] In an implementation, the method further comprises:

[0090] constructing a second optimization function, and applying the diffusion inversion model to invert and denoise the initial latent vector based on the empty text prompt, the text prompt and the second optimization function, to optimize the latent bias vector;

[0091] when the latent bias vector is optimal, determining the output vector of the diffusion inversion model as the target latent vector

[0092] In short, the text-guided diffusion model may have the problem of too strong conditional text constraints resulting in unnatural generated results. The Classifier-Free Guidance (CFG) technology is adopted in this embodiment, and the noise prediction of each step is composed of the conditional (text prompt) and non-conditional (empty text prompt) parts allocated by the guidance ratio λ. The method proposed in this embodiment eliminates the need to store the optimized empty text embedding for future use due to the accuracy of the inversion trajectory. By introducing the empty text prompt into the generation process, the editability of the model is significantly enhanced, which is very beneficial to downstream tasks such as image editing.

[0093] For the diffusion inversion model using the empty text prompt and the text prompt, this embodiment constructs a second optimization function corresponding to the diffusion inversion model. In the process of inverting and denoising the initial latent vector based on the empty text prompt and the text prompt by the diffusion inversion model, the latent bias vector in the diffusion inversion model is optimized based on the second optimization function to obtain the optimal latent bias vector, i.e. the optimal initial noise, so that the target latent vector obtained by the diffusion inversion model based on the optimal latent bias vector, the text prompt and the empty text prompt after inverting and denoising the initial latent vector can reconstruct an image with better accuracy.

[0094] In an implementation manner, the mathematical expression of adopting the diffusion inversion model to invert and denoise the initial latent vector based on the empty text prompt and the text prompt is:

[0095]

[0096]

[0097] wherein t is a time step, is a latent vector generated by the initial latent vector at the time step t, is a latent bias vector in the inversion process, is a latent bias vector in the denoising process.

[0098] Based on the above mathematical expression of adopting the diffusion inversion model to invert and denoise the initial latent vector based on the empty text prompt and the text prompt, the second optimization function comprises:

[0099]

[0100]

[0101]

[0102]

[0103] wherein, α t is a parameter predefined in the diffusion inversion model or the denoising diffusion implicit model, is an empty text prompt, and C is a text prompt, is a pre-trained UNet model, and λ>1.

[0104] As Figure 1 shown, the method further comprises the following steps:

[0105] Step S300, adopting a decoder to decode the target latent vector to determine a target image.

[0106] Briefly, the low-dimensional target latent vector located in the latent space is decoded and reconstructed by the decoder to obtain the target image corresponding to the initial image. In the present embodiment, the decoder of the vector quantization variational autoencoder is adopted.

[0107] In an implementation manner, before inverting and denoising the initial latent vector, the method further comprises:

[0108] Parameter freezing is performed on the encoder and the decoder;

[0109] The loss function is constructed based on a peak signal-to-noise ratio, structural similarity, and perceptual similarity.

[0110] The initial latent vector is optimized by using the encoder, the decoder, and the loss function after the parameters are frozen.

[0111] As shown in Figure 3 The encoder and the decoder in the embodiment are both implemented based on a vector quantization variational autoencoder. Since the reconstruction capability of the vector quantization variational autoencoder still has room for improvement, the peak signal-to-noise ratio (PSNR) of the image reconstructed by the vector quantization variational autoencoder ranges from 25 to 31 dB. For some images with rich textures or containing text content, the reconstruction performance can be even worse. In order to improve the accuracy of the reconstructed image, the embodiment proposes image information projection (IIP). During the projection process, the parameters of the encoder E and the decoder D are frozen, that is, the parameters in the encoder E and the decoder D are fixed and no longer participate in training; meanwhile, considering the need to maintain the consistency of the color, texture, and structure of the original image and the reconstructed image, the embodiment adopts a combination of the peak signal-to-noise ratio, structural similarity (SSIM), and perceptual similarity (LPIP) as the loss function; the initial latent vector of the image is trained based on the encoder, the decoder, and the loss function after the parameters are frozen (x0), more image information is projected into the latent space, and then the diffusion inversion model is used to perform inversion and denoising based on the optimized initial latent vector to determine the target latent vector, so as to obtain a target image with higher accuracy. z0

[0112] In an implementation manner, the optimization of the initial latent vector by using the encoder, the decoder, and the loss function after the parameters are frozen comprises:

[0113] Initialization The optimized initial latent vector is represented as x0, the initial image is represented as x0, the initial latent vector is represented as z0, and the encoder is represented as E.

[0114] The optimization of the initial latent vector is performed on

[0115]

[0116] wherein, the decoder is represented as D, the loss function is represented as L;

[0117]

[0118] wherein, the peak signal-to-noise ratio loss is represented as LPSNR, ​​as a structural similarity loss, as a perceptual similarity loss.

[0119] Through experiments, the image information projection method proposed in the embodiment can improve the peak signal-to-noise ratio of the reconstructed image by 2dB to 5dB using a small amount of existing time (about 10 seconds), so that the reconstruction quality of the image exceeds that of the vector quantization variational autoencoder.

[0120] Through experiments on different images, the image inversion method based on the diffusion model proposed in the embodiment has the following advantages: (1) by modeling the deviation of the latent vector in the adjacent inversion step of the denoising diffusion implicit model, the optimal initial noise is obtained without introducing additional parameters by optimizing the deviation in each step, which improves the image reconstruction accuracy and ensures the stability of the reconstruction effect; (2) the introduction of the empty text prompt and the text prompt at the same time enhances the editability of the model and is beneficial to the downstream task of image editing; (3) by using the combination of the peak signal-to-noise ratio, the structural similarity and the perceptual similarity to construct the loss function to optimize the initial latent vector, the method projects the original image information into the initial noise, so that the reconstruction accuracy of the method is no longer limited by the performance upper bound caused by the mismatch between the vector quantization variational autoencoder and the inversion process, and better reconstruction effect is achieved.

[0121] Based on the above embodiment, the application further provides an image inversion device based on a diffusion model, as shown in Figure 4 The device comprises:

[0122] An image encoding module 01 is configured to input an initial image into an encoder to determine an initial latent vector corresponding to the initial image.

[0123] An inversion and denoising module 02 is configured to obtain a text prompt, and use a diffusion inversion model to invert and denoise the initial latent vector based on the text prompt to determine a target latent vector, wherein the diffusion inversion model is realized by modeling the noise in the inversion and denoising process of a denoising diffusion implicit model using a latent deviation vector, and the denoising diffusion implicit model is realized based on a UNet model.

[0124] An image decoding module 03 is configured to use a decoder to decode the target latent vector to determine a target image.

[0125] Based on the above embodiment, the application further provides a terminal, and the principle block diagram thereof can be as shown in Figure 5The terminal includes a processor, a memory, a network interface, and a display screen connected through a system bus. The processor of the terminal is configured to provide computing and control capabilities. The memory of the terminal includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for running the operating system and the computer program in the non-volatile storage medium. The network interface of the terminal is configured to communicate with external terminals through a network connection. The computer program is executed by the processor to implement the image inversion method based on the diffusion model. The display screen of the terminal can be a liquid crystal display screen or an electronic ink display screen.

[0126] Those skilled in the art can understand that Figure 5 The principle block diagram shown in the figure is only a block diagram of part of the structure related to the present application, and does not constitute a limitation on the terminal to which the present application is applied. The specific terminal can include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0127] In an implementation manner, the memory of the terminal stores more than one program, and is configured to execute the more than one program by more than one processor, including instructions for performing the image inversion method based on the diffusion model.

[0128] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments. Any reference to memory, storage, database or other medium used in the embodiments provided by the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0129] To sum up, the application discloses an image inversion method and device based on a diffusion model, a terminal and a storage medium, the method inputs an initial image into an encoder to determine an initial latent vector corresponding to the initial image; obtains a text prompt, and uses a diffusion inversion model to perform inversion and denoising on the initial latent vector based on the text prompt to determine a target latent vector, wherein the diffusion inversion model is realized by modeling noise in the inversion and denoising process of a denoising diffusion implicit model using a latent bias vector, and the denoising diffusion implicit model is realized based on a UNet model; and a decoder is used to decode the target latent vector to determine a target image. The application models the bias between two adjacent latent vectors by using a latent bias vector, can obtain an optimal initial noise without introducing additional parameters, and effectively solves the problem that existing inversion methods based on the diffusion model cannot guarantee high-quality and high-accuracy reconstruction of an image while achieving low computational overhead.

[0130] It should be understood that the application is not limited to the above examples, and can be improved or changed according to the above description for those of ordinary skill in the art, and all these improvements and changes shall belong to the protection scope of the appended claims of the application.

Claims

1. A method of image inversion based on a diffusion model, characterized in that, The method comprises: inputting an initial image into an encoder to determine an initial latent vector corresponding to the initial image; obtaining a text prompt, and using a diffusion inversion model to perform inversion and denoising on the initial latent vector based on the text prompt to determine a target latent vector, wherein the diffusion inversion model is implemented by using a latent bias vector to model noise in the inversion and denoising process of a denoising diffusion implicit model, and the denoising diffusion implicit model is implemented based on a UNet model; using a decoder to decode the target latent vector to determine a target image; the diffusion inversion model based on the text prompt to perform inversion and denoising on the initial latent vector to determine a target latent vector, comprising: constructing a first optimization function or a second optimization function corresponding to the diffusion inversion model; when the first optimization function is constructed, using the diffusion inversion model to perform inversion and denoising on the initial latent vector according to the text prompt and the first optimization function to optimize the latent bias vector; when the latent bias vector is optimal, determining that the latent vector output by the diffusion inversion model is the target latent vector; the first optimization function comprises: ， wherein, is the time step, is the latent bias vector during the inversion process, is the latent bias vector during the denoising process, , , , is a parameter predefined by the diffusion inversion model at the time step, is a parameter predefined by the diffusion inversion model at the time step, is a text prompt, is the pre-trained UNet model, is the latent vector after inverting the initial latent vector for t time steps.

2. The image inversion method based on diffusion model according to claim 1, characterized in that, the method further comprises: when the second optimization function is constructed, using the diffusion inversion model to perform inversion and denoising on the initial latent vector based on an empty text prompt, the text prompt, and the second optimization function to optimize the latent bias vector; when the latent bias vector is optimal, determining that the output vector of the diffusion inversion model is the target latent vector.

3. The image inversion method based on diffusion model according to claim 2, characterized in that, the second optimization function comprises: ， wherein, is the time step, is the latent bias vector during the inversion process, is the latent bias vector during the denoising process, ， ， ， wherein, , , is the empty text prompt, is the pre-trained UNet model, , is the latent vector after t time steps of unrolling the initial latent vector, is the latent vector after t time steps of unrolling the initial latent vector. is the latent vector after t time steps of unrolling the initial latent vector.

4. The image inversion method based on diffusion model according to claim 1, characterized in that, before performing inversion and denoising on the initial latent vector, further comprising: parameter freezing the encoder and the decoder; constructing a loss function based on a peak signal-to-noise ratio, a structural similarity, and a perceptual similarity; optimizing the initial latent vector using the parameter-frozen encoder, the decoder, and the loss function.

5. The image inversion method based on diffusion model according to claim 4, characterized in that, the optimization of the initial latent vector using the parameter-frozen encoder, the decoder, and the loss function comprises: Initialization , , denotes the optimized initial latent vector, is the initial image, denotes the initial latent vector, is the encoder; Optimizing for performance: ， wherein, for the decoder, is the loss function; ， wherein, is a peak signal-to-noise ratio loss, is a structural similarity loss, is a perceptual similarity loss.

6. An image inversion apparatus based on a diffusion model, characterized by, The device is used to implement the diffusion model-based image inversion method as claimed in any one of claims 1-5, and the device comprises: An image encoding module is configured to input an initial image into an encoder to determine an initial latent vector corresponding to the initial image. An inversion and denoising module is configured to obtain a text prompt, and use a diffusion inversion model to perform inversion and denoising on the initial latent vector based on the text prompt to determine a target latent vector, wherein the diffusion inversion model is implemented by using a latent bias vector to model noise in the inversion and denoising process of a denoising diffusion implicit model, and the denoising diffusion implicit model is implemented based on a UNet model. An image decoding module is configured to use a decoder to decode the target latent vector to determine a target image.

7. A terminal, characterized by comprising: The terminal comprises a memory and one or more processors; the memory stores one or more programs; the programs contain instructions for executing the diffusion model-based image inversion method as claimed in any one of claims 1-5; and the processors are configured to execute the programs.

8. A computer-readable storage medium storing a plurality of instructions, characterized in that, The instructions are adapted to be loaded and executed by a processor to implement the steps of the image inversion method based on the diffusion model according to any one of claims 1-5.

Images