A controllable image generation method and system based on text instruction guidance

Abstract

The application discloses a controllable image generation method and system based on text instruction guidance, which effectively balances user friendliness and editing performance by combining the advantages of multi-modal large language model (MLLM) and diffusion model. Specifically, the application uses BLIP and fine-tuned GPT to parse user instructions, optimizes empty text embedding to adjust the inversion of noise trajectory, and injects cross-attention and self-attention control to realize natural language instruction driven controllable generation. The application can effectively cooperate multi-modal large language model and diffusion model, and has excellent performance in background fidelity and text alignment, etc. The application provides a new idea for improving the user friendliness and generation quality of controllable image generation, and further promotes the application of multi-modal generation technology in the field of digital media.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computer vision multi-modal generation, and relates to a controllable image generation method and system based on text instruction guidance, in particular to a controllable image generation method and system based on text instruction guidance of multi-modal model instruction analysis, optimized empty text embedding and attention injection. BACKGROUND

[0002] In the field of artificial intelligence generated content (AIGC), improving the controllability of image generation is an important sub-task. Image controllable generation (or image editing) uses artificial intelligence to create and modify digital content, and is considered an important field of innovation and practical application. Unlike image generation, which creates new images with minimal input, image editing involves changing the appearance, structure or content of an image, covering a range of changes from subtle adjustments to major transformations. Image editing is fundamental in various fields such as digital media, artistic design and scientific research, where changing visual content is crucial. The development of image editing reflects the progress of digital technology, from manual, labor-intensive processes using tools such as "Photoshop" to advanced digital technologies driven by learning-based algorithms.

[0003] However, traditional image editing techniques usually require users to provide detailed text descriptions or target object masks of images as input, greatly increasing the difficulty of practical application. In order to improve user friendliness and practicality, researchers have proposed an editing method based on natural language instructions. In this way, given an input image and a text editing instruction for the image (such as "remove the hat from the figure in the image"), the model can perform the corresponding editing operation and generate the edited image. This model does not require additional example images or masks, nor does it require a complete description of the input image or output image.

[0004] The main challenge of implementing instruction-based image editing with diffusion models is to build a "image-instruction-edited image" triplet dataset. This type of research generates data in two steps: first, given an image description (such as "a photo of a girl riding a horse"), use a fine-tuned GPT-3 to simultaneously generate an editing instruction (such as "let her ride a dragon") and an edited description (such as "a photo of a girl riding a dragon"); second, use Stable Diffusion and traditional image description-based editing algorithms to generate edited images, and finally collect a large-scale training image pair. Then train the diffusion model in a fully supervised manner, taking into account the conditional constraints of the input image and the instruction.

[0005] The existing instructional image editing methods have three main limitations: (1) They rely on static text encoders, which cannot accurately understand the context information of dynamic instructions, which can easily lead to over-editing or under-editing; (2) The training data is unreliable and is generated using traditional image description-based methods, which results in suboptimal model performance and requires a lot of computational resources for training; (3) The editing process is completely black box with poor interactivity. The editing needs, such as adjusting the intensity or size of the editing, cannot be fully expressed by human-provided instructions alone. Summary of the Invention

[0006] The purpose of this invention is to address the shortcomings of existing instruction-based image editing methods, which are difficult to implement, by fully utilizing the advantages of multimodal large models and diffusion models, and to provide a controllable image generation method and system based on multimodal model parsing of instructions, optimized empty text embedding, and attention injection guided by text instructions.

[0007] The technical solution adopted by the method of the present invention is: a controllable image generation method based on text instruction guidance, comprising the following steps:

[0008] Step 1: Given the input image Use a multimodal model to annotate images and obtain text descriptions of the source images. P Combined with user input editing commands The large language model is used to output triplet type editing information (target image description). Editing task types With local editing objects The local edit object is a tuple containing objects from the source and target image descriptions, respectively.

[0009] Step 2: Use noise and unconditional embedding To edit real input images; where weights are used without classifier guidance. A central noise trajectory is obtained by performing DDIM inversion. Its endpoint is ; For time steps;

[0010] Step 3: Simultaneously add text descriptions to the source image. P and target image description An iterative diffusion process is performed, in which attention-based operations, including self-attention map control and cross-attention map control, are applied at each step according to the type of editing task required, and the edited image is output. .

[0011] As preferred, the large language model in step 1 is a trained model; the training dataset used in the training process is a five-tuple dataset: (1) source image description, (2) editing instruction, (3) target image description, (4) editing type, (5) local editing object; wherein the source image description is a real picture description sampled from the dataset; using these data, the large language model in a single round is fine-tuned using the default training parameters.

[0012] As preferred, in step 2, the unconditional embedding is an optimized unconditional embedding.

[0013] First, define the empty text embedding , and the guide scale is , then the noise prediction without classifier guidance is defined as:

[0014]

[0015] wherein, is a noise estimator, is the noise vector at time step , and is the text embedding condition.

[0016] Then set the weight in the noise prediction without classifier guidance to A, and perform DDIM inversion to obtain a central noise trajectory , and the endpoint is ; then take this latent vector as the starting point, and perform step-by-step denoising, set the weight in the noise prediction without classifier guidance to B, convert the empty text embedding into a trainable embedding feature, and then optimize at each time step during the sampling process to reduce the difference between the sampling result and the original image; the objective function of the optimization process is:

[0017]

[0018] wherein, represents applying DDIM sampling using , the unconditional embedding , and the conditional embedding .

[0019] The DDIM inversion is:

[0020]

[0021] The DDIM sampling is:

[0022]

[0023] wherein, are a series of preset hyperparameters, A, B are preset threshold values.

[0024] As preferred, in step 3, the cross-attention map control is fixed internal randomness, first in each step, the noise vector and respectively perform sampling calculation to obtain attention map and ; then input the two into the editing function to obtain ; using replace , and then sample calculation to obtain ; finally, according to the automatically generated mask, further optimize the local editing area sampling; after steps, obtain , and output the edited image after decoding .

[0025] As preferred, avoid over-restricting geometric structure by adding timestamp soft constraint; the editing function is defined as:

[0026]

[0027] wherein, is a timestamp parameter, used to determine the step to start applying injection.

[0028] As preferred, add new segment tokens to the text description to achieve more detailed or global modification, first apply attention injection to the common tokens in the two prompts, then use the alignment function , use dynamic programming algorithm to solve a longest common subsequence problem, realize receiving token index from target image description , and output the corresponding token index in , if there is no match, output None; the editing function is defined as:

[0029]

[0030] wherein, index corresponds to pixel value, corresponds to text token.

[0031] As preferred, use a scaling factor to scale the attention map of the specified segment token , so as to produce stronger or weaker effect, and the rest of the attention map remains unchanged; the editing function is defined as:

[0032]

[0033] wherein the parameters , using the scaling factor can achieve fine intuitive control of the generation effect; the index corresponds to the pixel value, corresponds to the text token.

[0034] As a preferred, the mask of the edited part is approximated by the cross-attention map of the specific object, and the modification is constrained to apply only in the local area; for calculating the mask at the time step , first calculate the average attention map of the original word and the average attention map of the new word respectively, wherein denotes the average on step ; then apply threshold processing to generate a binary mask , wherein k is the threshold; take the final mask as the union of the two binary masks; finally, use the mask to constrain the editing area, mix the noise vector and to obtain the final , denotes element-wise multiplication.

[0035] As a preferred, the self-attention map control, the specific number of injections changes according to the editing type and user demand, so as to adapt to diversified editing needs.

[0036] The technical scheme adopted by the system of the present application is: a controllable image generation system based on text instruction guidance, comprising:

[0037] one or more processors;

[0038] a storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the controllable image generation method based on text instruction guidance.

[0039] Compared with the prior art, the beneficial effects of the present application include:

[0040] (1) The present application uses default training parameters to fine-tune a single round of large language model. Benefiting from the rich knowledge and generalization ability of the large language model, the fine-tuned model can generate a large amount and diversity of editing information set, while maintaining the correspondence between the target image editing information and the text instruction;

[0041] (2) In the prior art, due to the addition of a non-classifier guide, directly performing the DDIM inversion process will not only cause visual artifacts, but also the obtained noise vector may not conform to a Gaussian distribution. The application reduces the error introduced by optimizing the empty text embedding;

[0042] (3) The application generates a mask of the edited part by a cross-attention map of a specific object, and restricts the modification to be applied only in the local area;

[0043] (4) Based on the advantages of the multi-modal large model and the diffusion model, the application realizes controllable generation of instructed images and extracts rich editing information. By using empty text embedding optimization inversion and attention map control, the application realizes an interactive image controllable generation framework without training, improves the image generation quality, and reduces the image background loss. Finally, the application can be applied to actual industrial production sites to ensure the safe and reliable operation of production and the pursuit of high-quality products. BRIEF DESCRIPTION OF DRAWINGS

[0044] The technical solutions of the application are further described below using examples and specific embodiments. In addition, some drawings are also used in the process of describing the technical solutions. For those skilled in the art, other drawings and the intention of the application can also be obtained from these drawings without creative labor.

[0045] Figure 1 is a method flowchart of an embodiment of the application;

[0046] Figure 2 is a diffusion step flowchart of the cross-attention map control in the embodiment of the application. DETAILED DESCRIPTION

[0047] In order to facilitate those skilled in the art to understand and implement the application, the application is further described in detail below in combination with the drawings and examples. It should be understood that the implementation examples described herein are only used to illustrate and explain the application, and are not used to limit the application.

[0048] See Figure 1 The text instruction guided controllable image generation method provided by the embodiment includes the following steps:

[0049] Step 1: text instruction analysis;

[0050] Given an input image , use a multi-modal model to label the image to obtain a source image text description P ; combine user input editing instructions , and use a large language model to output a triple type editing information (target image description , editing task type with local editing objects ); where the local editing object is a two-tuple containing the object in the source image description and the object in the target image description, respectively;

[0051] In an embodiment, this step is implemented by the following sub-steps:

[0052] 1.1, Source image description generation: Image captioning is a basic task in visual language understanding, where a model predicts a textual information title for a given input image. Inspired by the generation capabilities of various multi-modal large models nowadays, the present invention first leverages the capability of the visual language model BLIP2 to complete image description generation, accurately extracting the visual information of the input image as a textual description. Specifically, given an input image , thanks to the image understanding capability of BLIP, the source image text description required for the subsequent steps can be obtained .

[0053] 1.2, Operating in the text domain, in order to enable the large language model to output rich and accurate editing information to meet the editing needs of the subsequent modules, it is first necessary to prepare the training data set and fine-tune the large language model.

[0054] The model used in this embodiment is trained by fine-tuning GPT4o-mini on a relatively small human-written editing five-tuple data set: (1) source image description, (2) editing instruction, (3) target image description, (4) editing type, (5) local editing object. The present invention samples more than 100 real image descriptions from the LAION-Aesthetics V2 6.5+ data set, and manually writes editing instructions, target image descriptions, editing task types and local editing objects.

[0055] The large language model is responsible for outputting a triple - target image description, editing type and local editing object. The target image description is the text description modified based on the source image description using the dynamic editing instructions provided by the user, for example, providing the input title "photo of a man wearing a hat" and the editing instruction "make him a woman", the large language model should be able to generate a reasonable output title "photo of a woman wearing a hat" after appropriate modification. The editing task type is roughly divided into three types, object replacement, attribute modification and attribute reweighting. Object replacement, such as "male" replaced by "female" in the previous example, attribute modification involves image style migration, object attribute refinement (such as "photo of a man wearing a duckbill cap") and a wider range, while attribute reweighting involves flexible adjustment of attribute strength (designed for multi-round interactive editing for users). Local editing object is for local editing, and this information is obtained to avoid excessive modification of irrelevant areas. In this example, "male" becomes "female", which belongs to the category of local editing, and the local editing object is "(male, female)"; if this example is modified to "photo of a man wearing a hat at dusk", it belongs to the type of global editing, and the style of the entire picture should be modified, so the local editing object is "None".

[0056] Using these data, the GPT model is fine-tuned for a single round using default training parameters. Benefiting from the rich knowledge and generalization ability of GPT, the fine-tuned model can generate a large and diverse set of editing information while maintaining the correspondence between the target image editing information and the text instructions.

[0057] 1.3, editing information acquisition: in the actual editing information extraction stage, the large language model fine-tuned in step 1.2 is used to receive the source image text description obtained in step 1.1 source image description generation And the user input editing instruction , obtain the triple type editing information (target image description , editing task type And local editing object );

[0058] Step 2: empty text optimization inversion;

[0059] Edit the real input image using noise And unconditional embedding ; Among them, the DDIM inversion is carried out using the middle weight Without classifier guidance, a central noise trajectory is obtained, and the endpoint is ; is the time step;

[0060] In an embodiment, to amplify the effect of the conditional text-guided generation, a classifier-free guidance technique is employed, in which the noise prediction is also performed unconditionally, and then extrapolated with the conditional prediction. Specifically, define the empty text embedding , and the guidance scale is Then the classifier-free guided noise prediction is defined as:

[0061]

[0062] where is the noise estimator, is the noise vector at time step , and is the text embedding condition.

[0063] To enable meaningful editing, it is usually required to invert the real image to obtain the corresponding noise vector for sampling in the diffusion model reverse process. Since editing on the real image needs to be able to accurately reconstruct a given real image from the noise, the present application employs a deterministic DDIM inversion:

[0064]

[0065] The sampling process employs a deterministic DDIM sampling:

[0066]

[0067] where is a series of preset hyperparameters.

[0068] However, since the classifier-free guidance is added, directly performing the DDIM inversion process will not only cause visual artifacts, but the obtained noise vector may not conform to a Gaussian distribution. The latter reduces the editability, i.e. the ability to edit using a specific noise vector.

[0069] To this end, the present embodiment first sets the weight in the classifier-free guidance to 1 (i.e. equivalent to no unconditionally denoised output), and performs DDIM inversion to obtain a central noise trajectory , the endpoint of which is . Then, starting from this latent vector, step-by-step denoising is performed, and for each timestamp , a separate optimization is performed, with the goal of making the denoised result at each time as close as possible to the corresponding noise vector in the central noise trajectory, thereby reducing the difference between the sampling result and the original image, with the objective function as follows:

[0070]

[0071] in It is an intermediate result of optimization.

[0072] Specifically, the weights in the classifier-free guidance Set to 7.5 (a common default setting for meaningful editing), for each timestamp. Optimize different empty text embeddings and use the embedding from the previous step To initialize The process involves transforming the empty text embedding into a trainable embedding feature, and then using the objective function proposed above to sample the corresponding feature at each time step. Optimize:

[0073]

[0074] in, Indicates use Unconditional embedding and conditional embedding Apply the DDIM sampling procedure. Then noise can be used. and optimized unconditional embedding To edit the actual input image.

[0075] Step 3: Focus Attention;

[0076] Simultaneously, the source image text description P and target image description An iterative diffusion process is performed, in which attention-based operations, including self-attention map control and cross-attention map control, are applied at each step according to the type of editing task required, and the edited image is output. .

[0077] In one implementation, the cross-attention map control:

[0078] set up Represents a single step in the diffusion process The calculation, its output is a noisy image. Attention map ;make This indicates the use of additional attention maps in the diffusion step. Cover the original attention map However, it retains the operation of the value provided by the text prompt; Indicates the use of edit description The generated attention map; Defined as a general editing function, it receives the original image and the edited image during the generation process. An attention map is used as input, based on the edit type. Output the modified attention map .

[0079] The flowchart of the diffusion steps controlled by the cross-attention graph is as follows: Figure 2 As shown. An iterative diffusion process is performed simultaneously on both text descriptions, where an attention-based operation is applied at each step according to the desired editing task type. At each step, the noise vector is... and Perform sampling calculations separately and Obtain attention map and Input both The function obtains ;use replace Again Sampling calculation get Finally, the sampling of the local editing area is further optimized based on the automatically generated mask. After several steps, we obtain The decoded image is the edited image. Since the same prompt word can produce significantly different outputs under different random seeds, this scheme fixes the internal randomness.

[0080] The following section explains the specific editing operations under different editing types, and completes the following steps. The function is defined, and the implementation of local editing is introduced.

[0081] (1) Object Replacement: In this case, the user replaces the token of an object in the original description with the token of another object. For example, the original description is "a man next to a red bicycle," while the edited description is "a man next to a red car." The goal is to preserve the original composition while generating the content of the new description. To this end, in one implementation, the attention map of the source image is injected into the generation process of the edited description. It should be noted that the final composition of the image is determined in the early generation stage of the diffusion process, and directly injecting cross-attention maps may overly restrict the geometry, especially when large structural modifications are involved (such as changing "car" to "bicycle"). Therefore, this embodiment addresses this issue by adding a timestamp soft constraint:

[0082]

[0083] in, is a timestamp parameter, which is used to determine the step to start the injection. By limiting the number of injection steps, both the composition generation and the necessary geometric freedom for adapting to new hints can be guided. To control the constraints more flexibly, different numbers of injection timestamps can be assigned to different tokens in the text description.

[0084] (2) Attribute modification: In the editing type of attribute modification, the user can add new segment tokens to the text description to achieve more detailed or global modification, for example, the original description is "children playing by the river" and the edited description is "in the evening, children playing by the river". In order to preserve the common details, the invention only applies attention injection to the common tokens in the two hints. Specifically, using the alignment function , a longest common subsequence problem is solved using a dynamic programming algorithm to realize the token index from the target image description is received, and the corresponding token index is output in , and None is output if there is no match. The editing function is defined as follows:

[0085]

[0086] , the index corresponds to the pixel value, corresponds to the text token. Similarly, the timestamp can be set to control the number of diffusion steps of the applied injection, thereby realizing various functions such as stylization processing, object attribute specification, or global manipulation.

[0087] Token is the smallest unit of text processed by Tokenizer (tokenizer), or the "segment" of a word, which is the smallest semantic unit that cannot be further divided, also known as "word unit". Most of the time, a word is a token; compound words such as "waterfall" will be divided into 2 tokens: water, fall; punctuation marks will also be defined as tokens.

[0088] (3) Attribute reweighting: After one editing process, the user may want to strengthen or weaken the influence of a token on the edited image, and further edit the generated image interactively. For example, the target description is "a boy is smiling", and the user wants to increase or decrease the degree of the boy's smile. To achieve this operation, the invention uses a scaling factor to scale the attention map of the specified segment token , thereby producing a stronger or weaker effect, while the rest of the attention map remains unchanged:

[0089]

[0090] where the parameters Using the scaling factor, fine and intuitive control of the generated effect can be achieved.

[0091] The local editing: In common application scenarios, users often need to modify a specific object or region while preserving other details (such as the background). In an embodiment, the mask of the edited part is approximated by the cross-attention map of the specific object, and the modification is constrained to apply only in the local region. To calculate the mask at time step , first calculate the average attention map of the original word (at step ) and the average attention map of the new word , respectively. Then apply threshold processing to generate a binary mask:

[0092]

[0093] where is set. To support geometric modification of the object, the editing region should contain the outlines of both the original object and the newly edited object, so the final mask is the union of the two binary masks. Finally, use the mask to constrain the editing region, mix the noise vector and to get the final , denotes element-wise multiplication:

[0094]

[0095] In an embodiment, the self-attention map control: each cell in the self-attention map determines the first and the first The weight assigned to each spatial feature correlation. Similar to the cross-attention map, the invention finds that the self-attention map is associated with the image structure and different semantic regions. Compared with cross-attention, the self-attention map contains more spatial layout and shape details of the source image, and the weight information represents the correlation between pixels and pixels, so the original image structure information can be injected into the generated target image through the self-attention mechanism. In some scenes where the requirement for preserving the structure of the original image is high, in addition to cross-attention injection, additional self-attention injection at certain steps is more helpful to preserve the source content, and if only cross-attention injection is used, the shape details of the original object cannot be preserved; while in some scenes where the requirement for preserving the shape of the target is low, additional injection of a large number of self-attention maps may lead to excessive restriction of its geometric shape, resulting in editing failure. The invention regards the control of the self-attention map as an indispensable part of preserving the content of the source image, and the specific number of injection steps is changed according to the editing type and user demand, so as to adapt to diversified editing needs.

[0096] Through detection, the invention can effectively improve the controllable image generation effect, while ensuring the quality of the generated image while taking into account the user friendliness.

[0097] Finally, using the image controllable generation method of the invention, the present scheme also carries out a comparative experiment to further illustrate the superiority of the design of the invention. The specific content of the experiment is as follows:

[0098] The invention compares the quantitative image editing performance of various models on the benchmark dataset PIE-Bench according to the existing controllable generation method. The invention evaluates three editing methods, covering multiple categories of image editing types, including: InstructPix2Pix (IP2P), Prompt-to-Prompt (P2P), and PowerPaint (PPT). Among them, Prompt-to-Prompt (P2P) is a method based on image inversion technology, and the invention also tests two different inversion technologies for this method: DDIM Inversion and Null-Text Inversion.

[0099] The experimental indicators of this experiment focus on the preservation degree of the unedited area (background fidelity) and the matching degree of the edited area and the instruction (text alignment).

[0100] The final experimental data is shown in Table 1:

[0101] Table 1

[0102]

[0103] The experimental data prove the superiority of the present application in retaining the unedited area and ensuring the text alignment degree of the edited area. Although the controllable generation method based on image inpainting (such as PPT) can achieve high-quality background retention, it needs the user to provide the mask by himself, essentially because the dedicated inpainting model ignores the non-edited area outside the mask. Since it cannot pay attention to the original information of the area inside the mask, this kind of method cannot retain the original structure and texture of the target, and cannot complete the background replacement, style editing and other tasks, so the text alignment degree of the edited area is significantly lower than that of other methods. In contrast, the present application uses the instruction analysis module to balance user friendliness and description accuracy, uses the optimization-based inversion method and attention injection mechanism, local editing mechanism and the like to further improve the background fidelity and text alignment degree, and in the case of equivalent background retention degree based on the mask input inpainting method, it can also achieve the best controllable generation effect.

[0104] It should be understood that the above-described embodiments are part of the embodiments of the present application, rather than all the embodiments. In addition, the technical features in each embodiment or single embodiment provided by the present application can be combined with each other to form a feasible technical solution, and such combination is not restricted by the sequence of steps and / or structure mode, but should be based on the realization by the ordinary skilled in the art, when the combination of technical solutions appears contradictory or unachievable, it should be considered that such combination of technical solutions does not exist, nor within the protection scope of the present application.

[0105] It should be understood that the above description of the preferred embodiments is more detailed, and therefore should not be considered as a limitation on the scope of patent protection of the present application. The ordinary skilled in the art can make substitutions or modifications under the inspiration of the present application without departing from the scope of protection claimed by the present application, all of which fall within the scope of protection of the present application. The scope of protection of the present application should be subject to the appended claims.

Claims

1. A controllable image generation method based on text-instruction guidance, characterized in that, Includes the following steps: Step 1: Given the input image Use a multimodal model to annotate images and obtain text descriptions of the source images. P Combined with user input editing commands The large language model outputs triplet type editing information, which includes the target image description. Editing task types With local editing objects The local editing object is a tuple containing objects from both the source and target image descriptions. Step 2: Use noise and unconditional embedding To edit real input images; where weights are used without classifier guidance. A central noise trajectory is obtained by performing DDIM inversion. Its endpoint is ; For time steps; Step 3: Simultaneously add text descriptions to the source image. P and target image description An iterative diffusion process is performed, in which attention-based operations, including self-attention map control and cross-attention map control, are applied at each step according to the type of editing task required, and the edited image is output. ; The cross-attention map control, wherein the internal randomness is fixed, firstly, in each step, the noise vector is processed. and Attention maps are obtained by performing sampling calculations separately. and Then input both into the editing function. get ;use replace Then again Sampling calculation Finally, the sampling of the local editing area is further optimized based on the automatically generated mask; after... After several steps, we obtain The edited image is output after decoding. ; A mask for the edited portion is approximated using a cross-attention map of a specific object, constraining modifications to apply only to that local region; this is used to compute time steps. The mask at the location is first calculated by dividing the original words. Average attention map and new words Average attention map ,in Indicates in step Average; then apply thresholding to generate a binarized mask. , where k is the threshold; take the final mask. The result is the union of two binary masks; finally, this mask is used to constrain the editing region, and the noise vector is mixed. and Get the final , This indicates element-wise multiplication.

2. The controllable image generation method based on text instruction guidance according to claim 1, characterized in that: The large language model mentioned in step 1 is a trained model; the training dataset used in the training process is a five-tuple dataset: (1) source image description, (2) editing instructions, (3) target image description, (4) editing type, (5) local editing object; where the source image description is a description of a real image sampled from the dataset; using these data, the large language model is fine-tuned for a single round using default training parameters.

3. The controllable image generation method based on text instruction guidance according to claim 1, characterized in that: In step 2, the unconditional embedding This is the optimized unconditional embedding; First, define the empty text embedding. The guiding scale is Then, the noise prediction without classifier guidance is defined as: in, It is a noise estimator. For time step The noise vector at time step, For text embedding conditions; Then the weights in the classifier-free guidance Set to A, perform DDIM inversion to obtain a central noise trajectory. Its endpoint is Then, starting with this latent vector, progressive denoising is performed, removing the weights from the unclassifier-guided process. Set to B to embed empty text. This is transformed into a trainable embedding feature, and then during the sampling process, the corresponding feature at each time step is... Optimization is performed to reduce the difference between the sampled result and the original image; the objective function of the optimization process is: in, Indicates use Unconditional embedding and conditional embedding Apply DDIM sampling; The DDIM inversion is as follows: The DDIM sampling is as follows: in, A and B are a series of preset hyperparameters, where A and B are preset thresholds.

4. The controllable image generation method based on text instruction guidance according to claim 1, characterized in that: Overly restrictive geometry is avoided by adding timestamp-based soft constraints; the edit function is defined as: in, This is a timestamp parameter used to determine the steps to begin the injection process.

5. The controllable image generation method based on text instruction guidance according to claim 1, characterized in that: To add new fragment tokens to the text description for more granular or global modifications, first apply attention injection to the common token in both hints, then use an alignment function. The longest common subsequence problem is solved using dynamic programming, which enables the description of a target image. Receive token index, and in Output the corresponding token index; if no match is found, output None. The edit function is defined as follows: Among them, index Corresponding to pixel values, This corresponds to the text token.

6. The controllable image generation method based on text instruction guidance according to claim 1, characterized in that: Scaling of the specified fragment token using a scaling factor The attention map is adjusted to produce a stronger or weaker effect, while the other attention maps remain unchanged; The edit function is defined as follows: Among them, parameters Using this scaling factor allows for precise and intuitive control over the generated effects; index Corresponding to pixel values, This corresponds to the text token.

7. The controllable image generation method based on text instruction guidance according to any one of claims 1-6, characterized in that: The self-attention map control allows the injection quantity to vary depending on the editing type and user needs, thus adapting to diverse editing requirements.

8. A controllable image generation system based on text command guidance, characterized in that, include: One or more processors; A storage device for storing one or more programs, which, when executed by one or more processors, cause the one or more processors to implement the text instruction-guided controllable image generation method as described in any one of claims 1 to 7.

Images