Text-guided generated image editing method

Abstract

The invention relates to the technical field of generated image editing, and provides a text-guided generated image editing method, which comprises the following steps of: firstly, performing multiple reasoning on an image generated by a given stable diffusion model by utilizing a multi-modal pre-training model BLIP model to generate a plurality of corresponding detailed text descriptions; then, performing intelligent text editing on a description text needing to be edited by using ChatGPT, and modifying a specific visual concept so as to meet a specific editing task for guiding an image; then, a CLIP model is used for calculating text embedding vectors before and after editing, so that the average direction of text editing is obtained; then, the average direction of text editing is added to an original description text embedding vector, the corresponding relation between the text and the image is controlled based on a cross attention mechanism of a stable diffusion model so as to accurately control the text guiding editing process of the image, and based on the average semantic direction of multiple sentences, robustness is achieved, and high isolation is achieved.

Description

Technical Field

[0001] The present invention relates to the field of generative image editing technology, and more specifically, to a text-guided generative image editing method. Background Art

[0002] Stable Diffusion and its derivative models have attracted much attention for their ability to generate high-quality images based on text descriptions. As an advanced generative model, Stable Diffusion can convert the language information input by users into expected visual content by encoding text prompts. It has been widely used in art creation, advertising design, virtual reality, and medical imaging. However, although this technology has shown great potential in multimodal generation tasks, in actual applications, it is often difficult for users to generate images that fully meet their expectations in one go, which adds uncertainty and challenges to the creation process. Therefore, in addition to generating images, the model also needs to have the ability to fine-tune the generated images so that users can adjust and optimize the generated results according to actual needs.

[0003] Currently, a significant limitation of the Stable Diffusion model is its high sensitivity to input prompts. Even slight changes in input prompts can result in completely different image outputs. For example, changing an adjective or adjusting the scene description may completely change the style or content of the generated image. This phenomenon is particularly prominent in application scenarios that require high-precision control of image content. In addition, even when using the same random seed, the output results of the model may fluctuate greatly, resulting in a large gap between the generated image and expectations. This problem is particularly prominent in areas such as image creation, advertising design, and film production, because in these scenarios, users often want to precisely control the specific details of the image to achieve a high degree of consistency with creative intent.

[0004] This unpredictability and control difficulty of the output make the Stable Diffusion model face great challenges in practical applications, especially when users want to edit details. Existing models usually rely on input text descriptions and random seeds to generate images, but lack sufficient mechanisms to ensure that even when users adjust certain details, other irrelevant parts of the image will not be affected. In addition, the randomness and noise factors in the generation process also make the results difficult to reproduce, increasing the difficulty for users to control image generation.

[0005] In order to solve the above problems, a technical solution is now provided. Summary of the invention

[0006] In order to overcome the above-mentioned defects of the prior art, the present invention provides a text-guided generative image editing method, which identifies stable semantic directions based on the current multimodal large model and language large model, and injects the semantic directions into the model reasoning process, so that the image editing can be carried out accurately along the semantic trajectory specified by the user, thereby avoiding unpredictable changes.

[0007] To achieve the above object, the present invention provides the following technical solutions:

[0008] A text-guided generative image editing method comprises the following steps:

[0009] Step S1, inputting the original text description into a pre-trained stable diffusion model, wherein the stable diffusion model performs reasoning based on the original text description and generates a corresponding image;

[0010] Step S2, using the multimodal pre-trained model BLIP model to infer the image generated by the stable diffusion model, and generating an image description corresponding to the image as the original image description;

[0011] Step S3, using the language model to perform text editing on the original image description, modifying a specific visual concept, and generating an edited image description;

[0012] Step S4, using the CLIP model to calculate the text embedding vector of the original image description and the text embedding vector of the edited image description, and obtain an average direction vector of the text editing;

[0013] Step S5, fusing the average direction vector of the text editing with the text embedding vector of the original text description, and controlling the correspondence between text and image based on the cross attention mechanism of the stable diffusion model.

[0014] As a further solution of the present invention, in step S5, different weighting coefficients are designed for the fusion process of the text embedding vector of the original text description and the average direction vector of the text editing. The specific formula is:

[0015] e=αe src +βe edit ;

[0016] In the formula, e is the fused text embedding vector, e src is the text embedding vector describing the original text, e edit is the average direction vector of text editing, α is the weighting coefficient of the text embedding vector described by the original text, and β is the weighting coefficient of the average direction vector of text editing.

[0017] As a further solution of the present invention, in step S5, the cross attention mechanism based on the stable diffusion model adjusts the correspondence between the text and the image by controlling the attention map in the reasoning process, including setting a hyperparameter in the initial stage of reasoning according to the sampling step number, when the reasoning step number is less than the hyperparameter, replacing the current attention map with an attention map generated according to the original text description, when the reasoning step number is greater than the hyperparameter, no additional control operation is performed on the current attention map, and the formula is:

[0018]

[0019] In the formula, c t is the current attention map during reasoning, c (prev) is the attention map generated based on the original text description, c (after) is the attention map generated according to the edited image description, t is the current sampling step in the reasoning process, and t′ is the set hyperparameter.

[0020] As a further solution of the present invention, the stable diffusion model is further combined with a mask-guided image editing method, by setting a mask to mark the target modification area, and applying different processing strategies to the areas inside and outside the mask respectively;

[0021] The area inside the mask is modified according to the semantic direction, and the area outside the mask remains unchanged. The real image is used to predict the noisy image of the current time step. Based on the calculation results inside and outside the mask, the two parts are synthesized using a fusion formula.

[0022] As a further solution of the present invention, the noisy image at the current time step is subjected to noise addition processing, and the noise added image is further subjected to denoising processing.

[0023] Compared with the prior art, the text-guided image editing method of the present invention has the following beneficial effects:

[0024] The present invention can effectively explore and identify robust semantic change directions by adopting the multimodal large model BLIP and the language model ChatGPT, so that the text prompt and the generated image can maintain stronger consistency. This method can accurately control the editing of image content and avoid the problem of subtle text changes in traditional models leading to completely different image outputs. Therefore, when using the present invention, users can more accurately guide the image generation process, significantly improving the controllability and reliability of text-to-image conversion.

[0025] The present invention further enhances the control accuracy of the editing area by integrating semantic direction into the reasoning process of the model and introducing a cross-attention mechanism. This mechanism can not only ensure that the modified part of the generated image meets the user's needs, but also perform local fine editing without interfering with other parts. In addition, the present invention also optimizes the deviation problems that may exist in the existing multimodal large model, indicating that better semantic alignment can enhance the robustness and accuracy of semantic direction, thereby effectively reducing the unpredictability in the generation process; the mask-guided local editing scheme significantly improves the editing accuracy of the image, especially when local adjustment is required, and further expands the application scope and practicality of image editing. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 The present invention is a flowchart of a text-guided image editing method for generating images.

[0027] Figure 2 The present invention provides a flowchart of text-guided image editing in a text-guided generated image editing method.

[0028] In the figure, text prompt: text prompt; T2I model: text to image model; "A cat in the hand": "A cat in the hand"; Image caption: image caption; "Holding a cat in one's palm.": "Holding a cat in one's palm."; "A feline nestled in the grasp.": "A cat nestled in the grasp."; "Clasping a feline companion.": "Holding a feline companion."; "A kitty cradled within one'shold.": "Holding a kitten in one'shold."; difference value: difference; cat->dog semantic direction: cat->dog semantic direction; edit text prompt: edit text prompt; "Holding a dog in one'spalm.": "Holding a dog in one'spalm."; "A canine nestled in the grasp.": "A canine nestled in the grasp."; "Clasping a canine companion.": "Holding a canine companion."; "A puppy cradled withinone's hold.": "Holding a puppy in one's hold." DETAILED DESCRIPTION

[0029] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0030] Example 1

[0031] A text-guided image editing method, characterized in that it comprises the following steps:

[0032] Step S1, inputting the original text description into the pre-trained Stable Diffusion model, and the Stable Diffusion model performs reasoning according to the original text description and generates a corresponding image;

[0033] Step S2, using the multimodal pre-trained model BLIP model to infer the image generated by Stable Diffusion, and generating an image description corresponding to the image as the original image description;

[0034] Step S3, using the language model ChatGPT to perform text editing on the original image description, modifying specific visual concepts, and generating an edited image description to meet the specific editing task of the guiding image;

[0035] Step S4, using the CLIP model to calculate the text embedding vector of the original image description and the text embedding vector of the edited image description, and obtain an average direction vector of the text editing;

[0036] Step S5, fusing the average direction vector of the text editing with the text embedding vector of the original text description, and controlling the correspondence between text and image based on the cross-attention mechanism of the Stable Diffusion model to accurately control the text-guided editing process of the image.

[0037] In step S5 of the embodiment of the present invention, different weighting coefficients are designed for the fusion process of the text embedding vector of the original text description and the average direction vector of the text editing. The specific formula is:

[0038] e=αe src +βe edit ;

[0039] In the formula, e is the fused text embedding vector, e src is the text embedding vector describing the original text, e editis the average direction vector of text editing, α is the weighting coefficient of the text embedding vector described by the original text, and β is the weighting coefficient of the average direction vector of text editing.

[0040] In the process of processing text and image embedding by the Stable Diffusion model in the embodiment of the present invention, the cross-attention mechanism is the key to controlling the relationship between text and image layout. In order to ensure that the original image remains unchanged as a whole and only realizes semantic changes in specific areas, the attention map of the original text is used to guide the generation of the current edited image.

[0041] In step S5 of the embodiment of the present invention, based on the cross attention mechanism of the Stable Diffusion model, the correspondence between the text and the image is adjusted by controlling the attention map in the reasoning process, including setting a hyperparameter in the initial stage of reasoning according to the sampling step number, when the reasoning step number is less than the hyperparameter, the current attention map is replaced with an attention map generated according to the original text description, when the reasoning step number is greater than the hyperparameter, no additional control operation is performed on the current attention map, and the formula is:

[0042]

[0043] In the formula, c t is the current attention map during reasoning, c (prev) is the attention map generated based on the original text description, c (after) is the attention map generated according to the edited image description, t is the current sampling step in the reasoning process, and t′ is the set hyperparameter.

[0044] In the embodiment of the present invention, the Softmax function is used to convert the scaled attention score into a weight:

[0045]

[0046] In the formula, the query matrix Key Matrix and the value matrix Represent the three key components of the cross-attention mechanism in the i-th layer of the UNet architecture. Here, Represents the intermediate representation of UNet flattened, through ∈ θ Optimize. Matrix and are learnable projection matrices that enable the intermediate representation and the conditional input y to be transformed into a common feature space for attention-based processing.

[0047] The Satble Diffusion model in the embodiment of the present invention is further combined with the mask-guided image editing method, by setting a mask to mark the target modification area, and applying different processing strategies to the areas inside and outside the mask, respectively, wherein:

[0048] The area within the mask is modified according to the semantic direction, and the calculation formula is:

[0049]

[0050] In the formula, is the noisy image of the area within the mask at time step t-1, α t-1 is the noise scheduling parameter at time step t-1, which is used to control the intensity of the noise, x t is the noisy image at the current time step t, α t is the noise scheduling parameter at time step t, used to control the intensity of the noise, ∈ θ (x t ,t) is based on the current image x t and the noise of the prediction at time step t, σ t is the noise intensity parameter at time step t, ∈ t is the random noise at time step t, adding randomness to the image;

[0051] The area outside the mask remains unchanged, and the real image is used to predict the noisy image at the current time step. The calculation formula is:

[0052]

[0053] In the formula, The noisy image of the area outside the mask at time step t-1, is the average noise scheduling parameter at time step t, x0 is the real image, ∈ is the random noise, generated by the model or sampled from the data;

[0054] Based on the calculation results inside and outside the mask, the two parts are synthesized using the fusion formula. The specific formula is:

[0055]

[0056] In the formula, x′ t-1 is the image obtained at time step t-1, including the result of fusion of the inner and outer regions of the mask, m is the mask matrix, is the noisy image of the area outside the mask at time step t-1, is the noisy image of the area within the mask at time step t-1, and ⊙ is the element-by-element multiplication.

[0057] In the embodiment of the present invention, the noisy image at the current time step is subjected to noise addition processing to introduce an additional noise signal, and the formula is:

[0058]

[0059] In the formula, x t+1 is the noisy image at time step t+1, x t is the noisy image at time step t, β t is the noise parameter at time step t, which controls the proportion of noise added to the image during the generation process, ∈ t is the random noise at time step t;

[0060] The denoising process is further applied to the denoised image, and the formula is:

[0061]

[0062] In the formula, x t-1 is the denoised image at time step t-1, x t is the noisy image at time step t, α t is the noise scheduling parameter at time step t, ∈ θ (x t ,t) is based on the current image x t and the noise of the prediction at time step t, σ t is the noise intensity parameter at time step t, z t is the random noise introduced, is the average noise scheduling parameter at time step t, z t is the noise variable at time step t, usually a random variable with a standard normal distribution.

[0063] Example 2

[0064] like Figure 2The figure shows a text-guided image editing process diagram of a text-guided generative image editing method of the present invention. First, the original text description "A cat in the hand" is input into the pre-trained Stable Diffusion. Satble Diffusion completes model reasoning to generate an image for a given text description; secondly, the generated image is input into the multimodal large model BLIP for multiple times to complete the image description generation task to obtain the corresponding text description "Holding a cat in one's palm", "A feline nestled in the grasp", "Clasping a feline companion", "A kitty cradled within one's hold", etc. for the input image. The diversified text description can more fully express the information of the image; thirdly, the generated diversified text description is input into the API of ChatGPT. The text descriptions designed according to the corresponding tasks are "Holding a dog in one's palm", "A canine nestled in the grasp", "Clasping a canine companion", "A puppy cradledwithin one's hold" etc. to complete the image description editing; then, the original image description and the image description modified based on ChatGPT are respectively used to obtain a series of text embedding vectors based on the text encoder of CLIP, and the average value of the difference of a series of text embedding vectors is calculated to obtain a robust text editing direction vector; finally, this text editing direction vector is fused into the text embedding vector of the original image description according to certain rules, and the obtained text embedding vector is used to guide image generation, and an additional cross-attention map is introduced to control the robustness of image generation.

[0065] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0066] Finally: The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A text-guided image editing method, characterized in that: The following steps are involved: Step S1, inputting the original text description into a pre-trained stable diffusion model, wherein the stable diffusion model performs reasoning based on the original text description and generates a corresponding image; Step S2, using the multimodal pre-trained model BLIP model to infer the image generated by the stable diffusion model, and generating an image description corresponding to the image as the original image description; Step S3, using the language model to perform text editing on the original image description, modifying a specific visual concept, and generating an edited image description; Step S4, using the CLIP model to calculate the text embedding vector of the original image description and the text embedding vector of the edited image description, and obtain an average direction vector of the text editing; Step S5, fusing the average direction vector of the text editing with the text embedding vector of the original text description, and controlling the correspondence between text and image based on the cross attention mechanism of the stable diffusion model.

2. A text-guided image generation editing method according to claim 1, characterized in that: In step S5, different weighting coefficients are designed for the fusion process of the text embedding vector of the original text description and the average direction vector of the text editing. The specific formula is: e=αe src +βe edit ; In the formula, e is the fused text embedding vector, e src is the text embedding vector describing the original text, e edit is the average direction vector of text editing, α is the weighting coefficient of the text embedding vector described by the original text, and β is the weighting coefficient of the average direction vector of text editing.

3. A text-guided image editing method according to claim 1, characterized in that: In step S5, the cross attention mechanism based on the stable diffusion model adjusts the correspondence between the text and the image by controlling the attention map in the reasoning process, including setting a hyperparameter in the initial stage of reasoning according to the sampling step number. When the reasoning step number is less than the hyperparameter, the current attention map is replaced with an attention map generated according to the original text description. When the reasoning step number is greater than the hyperparameter, no additional control operation is performed on the current attention map. The formula is: In the formula, c t is the current attention map during reasoning, c (prev) is the attention map generated based on the original text description, c (after) is the attention map generated according to the edited image description, t is the current sampling step in the reasoning process, and t′ is the set hyperparameter.

4. The text-guided image generation editing method according to claim 1, characterized in that: The stable diffusion model is further combined with a mask-guided image editing method, by setting a mask to mark the target modification area, and applying different processing strategies to the areas inside and outside the mask respectively; The area inside the mask is modified according to the semantic direction, and the area outside the mask remains unchanged. The real image is used to predict the noisy image of the current time step. Based on the calculation results inside and outside the mask, the two parts are synthesized using a fusion formula.

5. A text-guided image generation editing method according to claim 1 or 4, characterized in that: The noisy image at the current time step is subjected to denoising, and the denoising image is further subjected to denoising.

Images