Image Editing Method Based on Prior Constraint Inversion Algorithm

Abstract

The present invention discloses an image editing method based on a priori constraint inversion algorithm. The method firstly requires a user to input original text, edited text and corresponding image, and uses a large language model LLM for preprocessing to obtain filtered original text, edited text, and key words of the edited text. Secondly, the preprocessed original text and edited text are combined with key words and corresponding images to obtain the edited image through a diffusion model; specifically, the method includes: using the preprocessed original text and corresponding image to input into the diffusion model to obtain the latent space encoding of the final time step; inputting the latent space encoding and the preprocessed original text and edited text into the diffusion model to generate the final edited image. In order to enhance the editing effect and utilize the inherent characteristics of the diffusion model, the present invention introduces a guided intensity selector to improve the diversity and quality of the edited image.

Description

Technical Field

[0001] The invention relates to the technical field of image editing, and in particular to an image editing method based on a priori constraint inversion algorithm. Background Art

[0002] With the development of diffusion models, especially the emergence of latent diffusion models (LDMs), the field of image generation has undergone revolutionary changes. Taking advantage of the excellent semantic understanding ability of pre-trained LDMs, researchers have successfully applied them to many downstream tasks, such as text-to-image generation, style transfer, and text-driven image editing. In particular, the application scenario of editing specific attributes of real images has been widely studied. Modifying the specific attributes of real images to generate images that meet user needs has shown a wide range of application scenarios in social media, film and television entertainment, and inversion technology is a very critical step.

[0003] Although image editing methods based on inversion technology have achieved remarkable results, artifacts may be generated when using target cues to edit images, and there are still certain challenges for real image editing tasks.

[0004] (1) How to balance the editability and reconstructability of latent space encoding?

[0005] Reconstructibility means that given the original text, the latent space encoding can faithfully reconstruct the image; editability means that given the edited text, the latent space encoding can generate an image that conforms to the text semantics. Currently, many works use the deterministic inversion technology of the denoising diffusion implicit model (DDIM) to solve this problem. However, the DDIM inversion technology has assumptions in practical applications that affect the quality of editing. How to balance editability and reconstruction based on the DDIM inversion technology to further improve the quality of edited images remains a challenge.

[0006] (2) Poor diversity of edited images

[0007] Although the classifier-free guidance technology widely used in Wenshengtu can improve the generation quality, it will also reduce the diversity. How to ensure the editing quality while improving the editing diversity is still a challenge. Summary of the invention

[0008] In view of the above problems, this paper proposes a new image editing method, which aims to improve the performance of diffusion models, especially Stable Diffusion (SD) models in image editing tasks. Through these innovations, the development of image editing technology will be promoted, and more efficient and accurate solutions will be provided for the application of AIGC technology in image editing, personalized content generation, virtual fitting and other fields.

[0009] A fixed point optimization method based on prior constraints is proposed in the inversion stage. This method reduces the side effects of the DDIM deterministic inversion method by introducing fixed point constraints, and uses prior constraints to enhance the generalization ability of the latent space encoding at each time step, thereby facilitating subsequent editing. This dual constraint ensures accurate reconstruction of the original image and balances the reconstruction ability and editing ability of the latent code.

[0010] In the editing stage, the present invention introduces a guided strength selector (GDS) to address the problem that the classifier-free guided technology in the diffusion model may lead to reduced diversity and quality of edited images. GDS can adaptively adjust the guided strength of each time step, aiming to improve the quality of generated images and the diversity of editing effects.

[0011] The image editing method based on the prior constraint inversion algorithm has the following steps:

[0012] Step (1): The user inputs the original text, edited text and corresponding image, which are preprocessed using the Large Language Model (LLM).

[0013] The user inputs the original text, edited text and the corresponding image. The original text and edited text entered by the user are input into the large language model (LLM) for preprocessing to obtain the filtered and optimized original text and edited text. The powerful semantic understanding ability of the large model is used to obtain the key words of this editing using the original text and edited text. The key words appearing in the edited text indicate the words that have been changed during the editing process and are used in the editing stage.

[0014] This step (2): the preprocessed original text and edited text are combined with the preprocessed key words and the corresponding image, and the edited image is obtained through the diffusion model SD.

[0015] Step (2.1) enters the inversion stage, using the preprocessed original text and the corresponding image input into the diffusion model to obtain the latent space encoding of the final time step.

[0016] Step (2.2) enters the editing stage, where the latent space encoding and the preprocessed original text and edited text are input into the diffusion model and combined with some mature editing tools to generate the final edited image.

[0017] In step (1), the original text and the edited text are input into the large language model to filter out some sensitive words to prevent the editing technology from affecting the legitimate rights and interests of others. The original text and the edited text are used at the same time to obtain the key words of this editing through the large language model.

[0018] In the step (2.1), the purpose is to use the original text obtained in step (1) and the corresponding single-instance image to obtain the latent space encoding of the final time step through a set number of iterations. To achieve this goal, the present invention adopts the widely used Stable Diffusion (SD) model. The SD model is an advanced open source model that has been trained on large-scale text-image pairs, and it has shown excellent performance in the field of image generation. The SD model consists of three parts: an SD encoder, a decoder, and a U-net network for predicting noise. The SD model combines the classifier-free guidance technology CFG. First, the input image Via SD encoder , to obtain the latent space encoding , the original text is passed through the CLIP text encoder to generate the embedded code of the input text Similarly, the edited text is passed through the CLIP text encoder to generate the embedded code of the input text ;Will , and the current time step As input the U-net network predicts the noise for the current time step , expressed by formula (1):

[0019] (1)

[0020] in, is the time step, U-net network for SD model, For time The latent space encoding at this time, the embedded encoding of the text will be used as a guide to generate the condition. Get the noise of the current time step with the current time step Use formula (2) to obtain the latent space encoding of the next time step , represents the noise schedule parameter associated with the current time step. However, formula (2) is an approximation of formula (3) in the actual inversion case. This approximation can provide certain effects when the number of iterations is large, but due to the irreversibility between forward inversion and backward generation (editing), the accuracy in the reconstruction process is insufficient. In particular, in the forward process, high-intensity guidance cannot be applied, which reduces the editability of the latent space encoding.

[0021] (2)

[0022] Using the fixed point technique, formula (2) is expressed as The function is shown in formula (3):

[0023] (3)

[0024] use As the initial optimized saddle point, combined with formula (3), the fixed point function is constructed The constraint loss function is shown in formula (4):

[0025] (4)

[0026] At the same time, in order to further improve the editing ability of latent space coding, reduce the coupling between the original text and the current latent space coding, and improve the quality of the edited image, the present invention further uses the diffusion model SD prior constraint as shown in formula (5):

[0027] (5)

[0028] Using the above two constraint loss functions The latent space encoding is optimized until the difference of the latent space encoding is less than the threshold, or the set number of optimizations is reached, and the final latent space encoding of the next time step is obtained. ,use Repeat the above process for the preset number of inversion steps to obtain the latent space encoding of the final time step .

[0029] The step (2.2) first encodes the embedded text and combines it with the latent space encoding Input into the U-net network in the SD model, obtain the image features as the query Q, and the key word as the key (K), calculate the cross attention map corresponding to the key word through the cross attention mechanism, and obtain the predicted noise and the cross attention map for the key word. According to the cross attention map at this time, determine whether this time step contains the semantics of the corresponding morpheme, and then obtain the guidance strength of this time step Since U-net contains multiple layers, corresponding to cross-attention maps of various sizes, a cross-attention map of size 16x16 was selected in the actual process.

[0030] Specifically, we get the cross attention graph corresponding to the key words Afterwards, the Laplace operator is used, as shown in formula (6):

[0031] (6)

[0032] Use this kernel to convolve with the cross attention map. The specific calculation method is as follows:

[0033] (7)

[0034] According to the pre-set threshold right Binarize and generate a mask ,judge Whether it contains a continuous area; if so, the intensity will be guided Set to 7.5, otherwise set to 1.0;

[0035] (8).

[0036] The latent space encoding Separate and edit text embedding codes , the empty text is encoded by the CLIP text encoder and then input into the diffusion model to obtain , ;use , , combined with the guiding strength Get the noise enhanced by the edited text , as shown in formula (9):

[0037] (9)

[0038] (10)

[0039] Will , Input into formula (10) to get the latent space encoding of the previous time step , repeat the above process until the pre-defined number of iterations. After the iteration, the latent space is encoded , the decoder outputs the edited picture.

[0040] Beneficial effects of the present invention:

[0041] In this paper, we propose an innovative training-free improvement method to address the challenges of SD models in image editing tasks, aiming to enhance text-driven image editing capabilities. We fully exploit the inherent potential of the SD model and the prior knowledge of the diffusion model, and introduce two key losses in the inversion phase: an iterative loss for fixed points and a loss for diffusion priors, so as to accurately adjust the noise predicted by the model at each time step. This not only ensures the accurate reconstruction of the original image, but also maintains the editability of the latent space encoding.

[0042] In the editing stage, in order to enhance the editing effect and utilize the inherent characteristics of the diffusion model, a guidance strength selector is introduced to adaptively adjust the guidance strength, further improving the diversity and quality of edited images.

[0043] The present invention can realize more accurate and higher quality image modification, meet the user's demand for personalized services, provide a new idea and method for image editing based on diffusion model, and bring more innovative possibilities to other related fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 It is a specific schematic diagram of the process of the present invention;

[0045] Figure 2 It is the process of loss-guided optimization of latent space encoding at a single time step in the present invention;

[0046] Figure 3 It is a schematic diagram of the optimization module in the present invention;

[0047] Figure 4 Schematic diagram of the experimental results of the present invention. DETAILED DESCRIPTION

[0048] The present invention will be further described below in conjunction with the accompanying drawings and actual cases.

[0049] The image editing method based on the prior constrained inversion algorithm is mainly divided into three stages: first, filtering is performed using a pre-trained large language model (LLM), then the latent space encoding is optimized through prior constraints, and finally the final latent space encoding is input into the editing branch, and the guidance strength of the time step is obtained through the guidance strength selector for editing, and finally the target edited image is obtained.

[0050] like Figure 1 As shown, an image editing method based on a priori constraint inversion algorithm, the steps are as follows:

[0051] Step (1): The text prompt entered by the user, such as "a cat riding a toy bicycle" and the target edit text "a dog riding a toy bicycle", is sent to LLM for text filtering to identify and remove any sensitive words that may cause controversy to avoid the editing results infringing on the rights of others (avoid). Set hyperparameters such as the number of iterations and the maximum number of optimizations. Complete the initialization of the network.

[0052] The preprocessed original text and edited text are combined with the preprocessed key words and the corresponding image, and the edited image is obtained through the diffusion model SD.

[0053] Step (2) enters the inversion stage to obtain the latent space encoding corresponding to “a cat riding a toy bicycle”.

[0054] Step (3) enters the editing stage, inputs the latent space encoding obtained in step (2) as well as the original text and the edited text into the editor to obtain the final edited image.

[0055] Specifically, in step (1), a large language model (LLM) is used as the core of the filtering module to preprocess the text content and ensure that it complies with laws, regulations and ethical standards, thereby preventing the generation of inappropriate content such as racial discrimination, anti-social and anti-human content, avoiding the images edited by the invention of this technology from infringing the legitimate rights and interests of others, and ensuring the positive social impact of the technology. Since image editing may generate some misleading or inappropriate content, such editing results may be maliciously used to damage personal reputation or infringe privacy. It is very important to add content filtering and compliance checks to the editing process to prevent any possible abuse. Afterwards, the filtered original text and the edited text are input into the large language model LLM to obtain the key words of this editing. Key words refer to words that need to be paid attention to during the editing process. For addition and replacement tasks, key words are words to be added or replaced. For example, the original text is "a cat" and the edited text is "a dog", then the key word is "dog"; for deletion tasks, key words are other entities. For example, the original text is "an orange cat and a dog" and the edited word is "an orange cat". At this time, the key word is cat.

[0056] The latent space encoding obtained in step (2) is as follows Figure 2 Shown: Image First, the encoder To obtain the latent space encoding . Use CLIP text encoder to generate embedded encoding of input text and use it as the condition for text-guided generation Input into the U-net structure of the large language model SD. The original text is fused with the features of the image through the cross-attention mechanism to predict the current noise. The latent space encoding of the next time step is obtained using formula (1): , which is taken as the saddle point of optimization.

[0057] (1)

[0058] As well as the latent space encoding and text embedding encoding of the previous time step, the prior loss and fixed point loss are calculated respectively, as shown in formulas (2) and (3), respectively.

[0059] (2)

[0060] (3)

[0061] Using the back propagation algorithm Optimize and repeat the above optimization process until the maximum number of optimizations is reached or the latent space encoding difference is less than the difference threshold set in step (1), and the latent space encoding of t time steps is obtained. Repeat the above process and finally get the latent space encoding , the modules used in this process are called optimization modules, such as Figure 3 shown.

[0062] In step (3), due to the use of inversion technology, the editing task becomes a generation task. In order to ensure the generation effect, the Stable Diffusion (SD) model widely uses the classifier-free guidance technology (CFG) in the field of text images to strengthen the role of editing text in generating editing images.

[0063] Experimental results show that in order to generate images that are semantically consistent with the target text, it is crucial to use high-intensity guidance. However, traditional generation methods often apply fixed-intensity guidance throughout the generation process, which may reduce diversity in the generation process. In addition, in the early stages of generation, too high a guidance intensity will affect the generation (editing) quality. Using high-intensity guidance will disrupt the distribution of images, which tends to generate images that best match the text semantics, making the image distribution more sharp.

[0064] To this end, the present invention proposes an adaptive mechanism of Guidance-Selector (GDS) to dynamically adjust the guidance intensity of each time step of a single instance image. , Target Edit Text Input into Unet, edit text as key, value, image as query. Calculate the cross attention map of size 16x16. The whole process can be expressed by formula (4), where formula (4) calculates the cross attention map containing all morphemes of the target edit text, and further obtains the attention map corresponding to the key word.

[0065] (4)

[0066] Using the Laplacian operator Use the Laplacian operator With Cross Attention Map Convolution is performed, and the specific calculation method is as follows as shown in formula (5):

[0067] (5)

[0068] is the kernel value. In practice, the kernel of formula (6) is used. Binarization is performed to generate a mask, and the function provided by the image processing library is used to determine whether the mask contains a continuous area. If it does, the intensity of the guide Set to 7.5, otherwise set to 1.0.

[0069] (6)

[0070] The latent space encoding , edit text embedding code, and empty text embedding code are input into the diffusion model to obtain , . use , Get the noise enhanced by the edited text , as shown in formula (7), where Indicates editing of embedded text. Represents empty text, = .Will , Input into formula (3) to get the latent space encoding of the previous time step , repeat the above process until the pre-defined number of iterations is reached, and finally the corresponding edited image is output.

[0071] (7)

[0072] (3) Obtain the latent space encoding of the T-1 time step , repeat the above process until you get , and finally output the final edited image through the SD model decoder

[0073] Implementation examples: Figure 4 The original text is a dog, and the edited text is a cat. The first row is the method of the present invention, and the rest are the methods of NMG (ICLR2024) and Direct (ICLR2024). The first column is the original image, the second column is the reconstructed image, and the third column is the edited image.

Claims

1. An image editing method based on a priori constraint inversion algorithm, characterized in that: The following steps are involved: Step 1: The user inputs the original text, edited text and corresponding image, and the large language model LLM is used for preprocessing; Step 2: Combine the preprocessed original text and edited text, the preprocessed key words and the corresponding image, and obtain the edited image through the diffusion model SD; the details are as follows: Step 2.1: In the inversion stage, the preprocessed original text and the corresponding image are input into the diffusion model to obtain the latent space encoding of the final time step; the specific implementation process is as follows: The SD model consists of three parts: an SD encoder, a decoder, and a U-net network for predicting noise; the SD model combines the classifier-free guidance technology CFG; The input image Image passes through the SD encoder ε(·) to obtain the latent space code z0 = ε(Image), and the original text passes through the CLIP text encoder to generate the embedded code f of the input text p Similarly, the edited text is passed through the CLIP text encoder to generate the embedded code of the input text f b ; z0,f p And the current time step t0 is used as the input U-net network to predict the noise ∈ of the current time step, which is expressed by formula (1): ∈=∈ θ (z t ,t,f p ) (1) Where t is the time step, ∈ θ (.) is the U-net network of the SD model, z t is the latent space encoding at time t, at which the text embedding encoding will serve as the guiding condition for generation; Get the noise ∈ of the current time step and the z of the current time step t Use formula (2) to obtain the latent space encoding z' of the next time step t+1 , C t,1 ,C t,2 ,a t represents the noise schedule parameters associated with the current time step; z’ t+1 =C t,1 *z t +C t,2 *∈ θ (z t ,t,f p ) (2) Using the fixed point technique, formula (2) is used as z' t+1 The function is shown in formula (3): f θ (z’ t+1 )=C t+1,2 *∈ θ (z’ t+1 ,t+1,f p )+C t+1,1 *z t . (3) Use z' t+1 As the initial optimized saddle point, combined with formula (3), the fixed point function f is constructed θ (z' t+1 ) is shown in formula (4): The diffusion model SD prior constraint is shown in formula (5): Using the above two constraint loss functions, z' t+1 The latent space encoding is optimized until the difference of the latent space encoding is less than the threshold, or the set number of optimizations is reached, and the final latent space encoding z for the next time step is obtained. t+1 , use z t+1 Repeat the above process for the preset number of inversion steps to obtain the latent space encoding z of the final time step T ; Step 2.2: In the editing stage, the latent space encoding, the preprocessed original text, and the edited text are input into the diffusion model to generate the final edited image. The specific implementation process is as follows: The embedded text encoding is combined with the latent space encoding z T Input into the U-net network in the SD model, obtain the image features as the query Q, the key word as the key K, calculate the cross attention map corresponding to the key word through the cross attention mechanism, and obtain the predicted noise and the cross attention map for the key word; According to the cross attention map, determine whether this time step contains the semantics of the corresponding morpheme, and then obtain the guidance strength w of this time step; The latent space encoding z T Separately and edit text embedding code f b , the empty text is encoded by the CLIP text encoder and input into the diffusion model, and we get ∈ edit ,∈ empty ; Utilize∈ edit ,∈ empty , combined with the guidance strength w, we get the noise ∈ opt , as shown in formula (6): ∈ opt =w∈ edit +(1-in) ∈ emmpty (6) z’ T+1 =C t,1 *z T +C t,2 *∈ opt (7) Will ∈ opt 、z T Input into formula (7) to get the latent space encoding z of the previous time step T-1 Repeat the above process until the pre-defined number of iterations. After the iteration, the latent space encoding z 0′ , the decoder outputs the edited image.

2. The image editing method based on the prior constraint inversion algorithm according to claim 1 is characterized in that: The specific implementation process of step 1 is as follows: The original text and edited text entered by the user are input into the large language model LLM for preprocessing to obtain the filtered and optimized original text and edited text. The semantic understanding ability of the large model is used to obtain the key words of this editing using the original text and edited text. The key words appearing in the edited text indicate the words that have been changed during the editing process.

3. The image editing method based on the prior constraint inversion algorithm according to claim 2 is characterized in that: The process of obtaining the guiding strength w is as follows: After obtaining the cross attention map I corresponding to the key word, the Laplacian operator is used, as shown in formula (8): Use this kernel to convolve with the cross attention map. The specific calculation method is as follows: According to the pre-set threshold Binarize L(x,y) to generate a mask Mask, and determine whether the Mask contains a continuous area; if so, set the guide strength w to 7.5, otherwise set it to 1.0;

Images