A text-driven multi-concept personalized image editing method

Abstract

The present invention discloses a text-driven multi-concept personalized image editing method, which first trains a concept model containing personalized targets to obtain a corresponding text embedding vector. Secondly, the text embedding vector is used to modify the source input text to obtain the modified text. Then, a stable diffusion model is used to load the trained concept model parameters containing personalized targets to obtain a diffusion model, and the source input text and the modified text are respectively input into the text encoder to obtain text embedding. The text embedding is input into the diffusion model to guide the denoising process, and the operation of replacing the attention map and the noise is performed during the denoising process. Finally, the iteration is repeated N times to obtain the target image, and then the real target image is obtained by decoding to complete the image editing. The present invention realizes the user's demand for multi-target personalized editing through text input, and expands the application scope of image editing in the AIGC field.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision, and in particular relates to a text-driven multi-concept personalized image editing method. Background Art

[0002] With the rapid development of AIGC (generative artificial intelligence) technology, large-scale text-to-image models, such as Stable Diffusion (a text-to-image model), Imagen (a text-to-image model), and DALL E-2 (a text-to-image model), have demonstrated remarkable generative capabilities. The rapid development of these technologies has enabled everyone to become an artist, creating their own unique artwork. However, while these models can generate stunning images, some even reaching the level of art, they lack a simple and efficient editing method for users, which is undoubtedly a shortcoming of this type of generative technology.

[0003] GLIDE (Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models) proposes a text-guided image editing method. However, it is worth noting that GLIDE requires the user to create a mask for the edited area. The mask creation process is tedious and user-unfriendly, especially when there are a large number of images to be edited, as it is both time-consuming and labor-intensive.

[0004] Prompt-to-Prompt Image Editing with Cross Attention Control (Prompt to Prompt Image Editing with Cross Attention Control) proposes a purely text-driven image editing method. It does not require the user to provide an editing area mask; users can edit specific targets simply by modifying text input, significantly lowering the barrier to entry for non-professionals. While Prompt to Prompt provides a highly efficient editing method, it cannot achieve personalized editing of the target, such as replacing the dog in an image with a designated pet dog.

[0005] Fine-tuning text-to-image diffusion models to achieve subject-driven generation. The emergence of Dreambooth (FineTuning Text-to-Image Diffusion Models for Subject-Driven Generation) and Lora (Low-Rank Adaptation of Large Language Models) has made personalized editing possible, as they can learn the concept of a specific target through training and generate it in different scenarios. Unfortunately, however, they cannot implement editing operations. That is, although users can generate the desired target in different scenarios, the background changes are uncontrollable, and users cannot generate a specific target in a specific scene.

[0006] To address the above issues, PhotoSwap (PHOTOSWAP: Personalized Subject Swapping in Images) proposes a target replacement method that combines Dreambooth and Prompt to Prompt to achieve personalized editing of specific targets in an image. However, PhotoSwap can only perform personalized editing on the same target in an image at the same time, and becomes powerless when personalized editing of multiple types of targets in an image is required. Personalized image editing of multiple targets is more in line with user habits and can better meet user needs, which is also a point that the existing technology urgently needs to overcome. Summary of the Invention

[0007] In order to solve the above technical problems, the present invention proposes a text-driven multi-concept personalized image editing method, which adopts a multi-concept learning method to learn multiple target features that need to be edited into the concepts of the model, and uses a text-driven approach to achieve the multi-concept personalized editing goal of the image.

[0008] In order to achieve the above object, the present invention adopts the following technical solutions:

[0009] The present invention provides a text-driven multi-concept personalized image editing method, which specifically includes the following steps:

[0010] Step 1: Train the concept model containing the personalized target and obtain the corresponding text embedding vector <new1> ， <new2>For guided image editing.

[0011] Step 2: Use the text embedding vector obtained in step 1 <new1>and <new2>, modify the source input text P 源 , obtain the modified text P 目标 .

[0012] Step 3: Use the StableDiffusion model to load the concept model parameters containing personalized targets trained in Step 1 to obtain a diffusion model. Input the source input text P 源 and the modified text P 目标 into the text encoder respectively to obtain text embeddings, and then input the obtained text embeddings into the diffusion model to guide the denoising process, and perform operations of replacing the attention map and noise during the denoising process.

[0013] Step 4: Repeat the operations in Step 3, iterate N times, and perform different operations at different stages from the 0 - λ (λ < N) times. Perform the operation of modifying and replacing the cross-attention map at the 0 - N1 (N1 < λ) times, perform the operation of modifying the self-attention map at the 0 - N2 (N2 < λ) steps, and perform the operation of replacing the result of the attention map at the 0 - N3 (N3 < λ) steps to obtain the target image.

[0014] Step 5: Decode the target image obtained in Step 4 to obtain the real target image, and complete the image editing.

[0015] A further improvement of the present invention is that: in the above Step 1, a model for training a concept containing personalized targets is obtained, and the corresponding text embedding vector <new1> ， <new2>, specifically including the following steps:

[0016] Step 1.1: Collect multiple images containing the same target as a training set, and write the training set paths of multiple targets into a .JSON file.

[0017] Step 1.2. Select text embeddings [new1] and [new2] to refer to different targets in step 1.1.

[0018] Step 1.3: Input the .JSON file in step 1.1 and the text embeddings of [new1] and [new2] in step 1.2 as input conditions into the multi-concept generation model for training, and obtain the concept model containing the personalized target and the text embedding vector. <new1> ， <new2>.

[0019] A further improvement of the present invention is that: the modified text P obtained in step 2 is 目标 The specific steps include:

[0020] Step 2.1: Determine the source input text P 源 , source input text P 源 The base class that should contain personalization targets.

[0021] Step 2.2: For the source input text P obtained in step 2.1 源 , using the text embedding vector obtained in step 1 <new1> ， <new2>Refers to the source input text P respectively 源 The corresponding target in the source input text P 源 With other embeddings unchanged, the modified text P containing the base class embedding is obtained 目标 .

[0022] A further improvement of the present invention is that the step 3 performs the operation of replacing the attention map and the noise, specifically comprising the following steps:

[0023] Step 3.1. Input source text P 源 and the modified text P 目标 At the same time, they are input into the U-net network model in the diffusion model and reasoning is performed separately.

[0024] Step 3.2: During the step-by-step denoising process of U-net, two results are generated at the same time due to the parallel reasoning in step 3.1. 源 The cross attention map and self attention map results in the denoising process are copied to the modified text P 目标 The corresponding positions of the two noise images with some noise removed are obtained.

[0025] Step 3.3: Keep the source input text P 源 The target area of ​​the cross attention map of the base class of the personalized target remains unchanged, and the noise in the rest of the area is replaced by the modified text P 目标 The corresponding area of ​​the noisy image.

[0026] A further improvement of the present invention is that the step 4 iterating N times specifically includes the following steps:

[0027] Step 4.1. In iterations 0-N1, perform the operation of modifying the cross attention map described in step 3.2.

[0028] Step 4.2: In iterations 0-N2, perform the operation of modifying the self-attention map described in step 3.2.

[0029] Step 4.3: In iterations 0-N3, perform the noise replacement operation described in step 3.3.

[0030] The beneficial effects of the present invention are:

[0031] 1. This paper innovatively proposes a text-driven multi-concept personalized image editing method. First, the personalized targets to be edited are jointly learned into the concepts of the diffusion model used. Then, by replacing the attention map of the editing target and its noise during the denoising process, the user can achieve multi-target personalized editing through text input, greatly expanding the application scope of image editing in the AIGC field.

[0032] 2. The present invention innovatively introduces a multi-concept learning method. Users only need to provide a small number of photos of personalized editing targets, and then through joint training, the features of multiple targets are learned into the concepts of the model. At the same time, different embedding vectors are set for different targets, so that users can complete the goal of multi-concept personalized editing by simply modifying the text, which greatly facilitates users to use the present invention to perform multi-concept personalized image editing operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 A flowchart of a text-driven multi-concept personalized image editing method provided in an embodiment;

[0034] Figure 2 A schematic diagram of a training set for "cat" provided in an embodiment;

[0035] Figure 3 A schematic diagram of a training set of "pot" provided in the embodiment;

[0036] Figure 4 A visualization diagram after exchanging cross-attention graphs provided in an embodiment;

[0037] Figure 5 A visualization diagram after noise replacement provided in an embodiment. DETAILED DESCRIPTION

[0038] The following diagrams illustrate embodiments of the present invention. For clarity, many practical details are further described in the following description. However, it should be understood that these practical details are not intended to limit the present invention. In other words, in some embodiments of the present invention, these practical details are not essential.

[0039] This paper proposes a text-driven, multi-concept personalized image editing method. This method incorporates multiple personalized target concepts into a diffusion model through training. It then achieves personalized editing of multiple objects in an image by replacing cross-attention maps, self-attention maps, and noise. This overcomes the limitation of previous models that only allow editing of single objects in an image. Furthermore, this method does not require additional masks, significantly facilitating user experience.

[0040] Example

[0041] like Figure 1 As shown, the embodiment of the present invention provides a text-driven multi-concept personalized image editing method, which specifically includes the following steps:

[0042] Step 1: Train the model containing the concept of personalized target and obtain the corresponding text embedding vector <new1> ， <new2>For guided image editing.

[0043] The specific steps include:

[0044] Step 1.1, collect multiple images containing the same target as the training set, and write the training set paths of multiple targets into a unified .JSON file, such as Figure 2 is the training set of "cat" used in this embodiment, Figure 3 is the training set of "pot" in the embodiment.

[0045] Step 1.2. Select text embeddings [new1] and [new2] to refer to different targets in step 1.1.

[0046] Step 1.3: Input the .JSON file in step 1.1 and the text embeddings of [new1] and [new2] in step 1.2 as input conditions into the multi-concept generation model Custom Diffusion (Multi-Concept Customization of Text-to-Image Diffusion) for training to obtain a concept model containing personalized targets and a text embedding vector. <new1> ， <new2>.

[0047] Step 2: Use the text embedding vector obtained in step 1 <new1>and <new2>, modify the source input text P 源 , get the modified text P 目标 .

[0048] The specific steps include:

[0049] Step 2.1: Determine the source input text P 源 ="Acat stands next to the pot", source input text P 源 Contains "cat" and "pot" for personalized targets.

[0050] Step 2.2: For the source input text P obtained in step 2.1 源 , using the text embedding vector obtained in step 1 <new1> ， <new2>Refers to the source input text P respectively 源 "cat" and "pot" in the source text P 源 When other embeddings remain unchanged, we get the value containing " <new1>cat”," <new2>pot" embedded modified text P 目标 ="A <new1>cat stands next to the <new2>pot".

[0051] Step 3: Use the Stable Diffusion v1.4 version as the base model, load the concept model parameters containing the personalized target trained in step 1 to obtain the diffusion model containing the personalized target concept (the diffusion models mentioned later are all diffusion models containing the personalized target concept), and input the source text P in step 2. 源 and the modified text P 目标 The text embeddings are input into the text encoder respectively, and then the obtained text embeddings are input into the diffusion model as conditions to guide the denoising process and replace the attention map and noise in the denoising process.

[0052] Specifically including the following steps:

[0053] Step 3.1. Input source text P 源 and the modified text P 目标 At the same time, it is input as text conditions into the U-net network model in the diffusion model and reasoning is performed separately.

[0054] Step 3.2: During the step-by-step denoising process of U-net, two results are generated at the same time due to the parallel reasoning in step 3.1. 源 The cross attention map and self attention map results in the denoising process are copied to the modified text P 目标 At the corresponding position of , we can get two noise images with some noise removed, such as Figure 4 Shown is a schematic diagram after replacing the cross attention map.

[0055] Step 3.3: Keep the source input text P 源 The key area of ​​the cross attention map of the base class of the personalized target remains unchanged, and the noise in the rest of the area is replaced by the modified text P 目标 The corresponding area of ​​the noisy image.

[0056] Step 4: Repeat the operation in step 3, iterate N times, perform the cross-attention map modification operation in time periods 0-N1, perform the self-attention map modification operation in time periods 0-N2, and replace the attention map results in time periods 0-N3 to obtain the target image.

[0057] The specific steps include:

[0058] Step 4.1: Perform the denoising process N times in total.

[0059] Step 4.2: Perform the operation of modifying the cross attention map described in step 3.2 for 0-N1 times.

[0060] Step 4.3: Perform the operation of modifying the self-attention map described in step 3.2 for 0-N2 times.

[0061] Step 4.4: Perform the noise replacement operation described in step 3.3 for 0-N3 times.

[0062] Step 5: Decode the target image obtained in step 4 to obtain the real target image.

[0063] Step 5.1: Use the VAE Decoder to decode the denoised image obtained in step 4 to obtain the edited image.

[0064] like Figure 5 The figure below is a schematic diagram of the final editing effect of this embodiment. As shown in the figure, the present invention successfully edited the words "cat" and "pot" in the image, and the features of the edited objects are similar to those in the training set. These results demonstrate the feasibility of the present invention and its good performance.

[0065] The embodiments of the present invention are described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Various modifications can be made within the scope of knowledge possessed by those skilled in the art without departing from the spirit of the present invention. Specific embodiments of the present invention will be described below with reference to the accompanying drawings. < / new1> < / new1> < / new1> < / new1> < / new1> < / new1> < / new1>

Claims

1. A text-driven multi-concept personalized image editing method, characterized in that: The following steps are involved: Step 1: Train the concept model containing the personalized target and obtain the corresponding text embedding vector <new1> ， <new2> Guided image editing, specifically including the following steps:< / new2> < / new1> Step 1.1: Collect multiple images containing the same target as a training set, and write the training set paths of multiple targets into a .JSON file; Step 1.2, select text embeddings [new1] and [new2] to refer to different targets in step 1.1; Step 1.3: Input the .JSON file and the text embeddings of [new1] and [new2] into the multi-concept generation model for training to obtain the concept model containing the personalized target and the text embedding vector. <new1> ， <new2> ；< / new2> < / new1> Step 2: Use text embedding vectors <new1>and <new2>, modify the source input text P 源 , get the modified text P 目标 , specifically including the following steps: Step 2.1: Determine the source input text P 源 , source input text P 源 Contains the base class for personalized targets; Step 2.2: For source input text P 源 , using text embedding vectors <new1> ， <new2>Refers to the source input text P respectively 源 The corresponding target in the source input text P 源 With other embeddings unchanged, the modified text P containing the base class embedding is obtained 目标 ; < / new1> Step 3: Use the stable diffusion model to load the trained concept model parameters containing personalized targets to obtain the diffusion model, and input the source text P 源 and the modified text P 目标 The text embedding is input into the text encoder to guide the denoising process, and the attention map and noise are replaced in the denoising process. The operation of replacing the attention map and the noise specifically includes the following steps: Step 3.

1. Input source text P 源 and the modified text P 目标 At the same time, they are input into the U-net network model in the diffusion model and reasoned separately; Step 3.2: During the stepwise denoising process of U-net, two results are generated at the same time; the source input text P 源 The cross attention map and self attention map results in the denoising process are copied to the modified text P 目标 At the corresponding positions of , two noise images with some noise removed are obtained; Step 3.3: Keep the source input text P 源 The target area of ​​the cross attention map of the base class of the personalized target remains unchanged, and the noise in the rest of the area is replaced by the modified text P 目标 The corresponding area of ​​the noise image; Step 4: Repeat the operation in step 3 for N times to obtain the target image; Step 5: Decode the target image to obtain the real target image and complete the image editing.

2. The text-driven multi-concept personalized image editing method according to claim 1, characterized in that: The step 4 iterating N times specifically includes the following steps: Step 4.1: In iterations 0-N1, perform the operation of modifying the cross attention map in step 3.2; Step 4.2: In the 0-N2 iterations, perform the operation of modifying the self-attention map in step 3.2; Step 4.3: In iterations 0-N3, perform the noise replacement operation in step 3.3.

Images