Multi-target image generation method based on attention differentiation and denoising process guidance

Abstract

The invention discloses a multi-target image generation method based on attention differentiation and denoising process guidance, which comprises the following steps of: 1) aiming at a given text prompt y comprising N target words and an initial implicit code zT, generating a cross attention graph CAT between each target word in the text prompt y and the zT by utilizing a stable diffusion model; 2) optimizing an initial implicit code zT by setting a loss function; 3) the stable diffusion model generates an optimal cross attention graph # imgabs0 # between each target word in the text prompt y and the optimized initial implicit code zT; 4) the stable diffusion model performs iterative denoising on the optimized initial implicit code zT and performs interpolation on the cross attention graph generated by a set time step; and decoding the hidden code z0 obtained after iteration denoising is finished to generate an image corresponding to the text prompt y. According to the method, the semantic matching between the generated image and the given text prompt is effectively improved.

Description

Technical Field

[0001] The invention belongs to the technical field of image generation, and relates to a multi-target image generation method based on differentiated attention and denoising process guidance. Background Art

[0002] The task of generating images from text aims to generate visually natural and semantically faithful images using text prompts. The traditional way of generating images is for designers to draw images by hand or use software based on text descriptions. In recent years, with the rapid development of artificial intelligence technology, various text-generated image technologies based on deep generative models have been widely used in many fields such as art design, digital development, and personalized customization. Compared with traditional methods, this image generation technology has made great progress in generation efficiency.

[0003] In the development history of deep generative models, many technical branches have emerged, such as variational autoencoders, adversarial generative networks, regularized flows, and diffusion models. Among them, the diffusion model has been widely discussed and applied in academia and industry due to its high quality and diverse generation of generated images. Among the diffusion model methods, the stable diffusion model stands out as an open source representative model of text-generated image models, and has led the trend of text-generated image technology based on deep learning.

[0004] However, even the most advanced stable diffusion models sometimes fail to generate images that fully match the given text prompts, especially when generating multiple objects. Figure 1 As shown in Figure 1, this weakness mainly manifests itself in the following aspects: (1) target missing, that is, one or more targets in the text prompt cannot be generated normally; (2) semantic leakage, that is, one target leaks its semantics to another target, resulting in a mixture of multiple targets in the generated image; (3) attribute error, that is, the model incorrectly binds the target to its attribute. Therefore, how to achieve seamless alignment between the generated image and the given text prompt is very critical for the application of the stable diffusion model technology.

[0005] At present, in the image generation task of multi-objective combination, in order to improve the consistency of images and texts generated by the stable diffusion model, the methods used are mainly divided into two categories: the first category is based on additional input to provide additional control conditions for generation, and the other is a method without additional input. Based on the above two types of methods, the defects of their respective background technologies are summarized as follows:

[0006] (1) Methods based on additional input

[0007] Methods based on additional inputs improve the controllability of the generation process and the image-text matching ability of the stable diffusion model by giving additional input information. In addition to text prompts, additional image layout information (such as the semantic segmentation map of the image, or the bounding box of the target) is usually introduced as input to restrict the position and size of the target in the image. Recently, there are also methods that can provide finer-grained control capabilities in the image space. Since such methods accept additional user input as control, the images they generate are more likely to meet the user's generation requirements, and the image-text matching is also higher. However, the input of additional control conditions requires manual annotation, which consumes a high time cost, so it is not applicable in many real applications.

[0008] (2) Methods without additional input

[0009] In the scenario where only text prompts are used as control conditions, in order to correctly output multiple targets in the generated image, some methods try to generate multiple targets separately and then splice the model outputs. Recently, more and more work has been devoted to generation control at the attention level. In the stable diffusion model, since the cross-attention reflects the area of ​​the text interaction image and the intensity of the interaction, direct control at the cross-attention level can change the expression intensity of each word in the text prompt, thereby improving the consistency of the text image. Guided by the loss at the attention level, some of these methods optimize the hidden code at each time step of the diffusion model denoising, while others only focus on the optimization of the initial noise. However, these methods are not comprehensive. The optimization of the hidden code at each time step is likely to result in the generation of out-of-distribution images, that is, the hidden code jumps out of the original distribution manifold of the model after over-optimization, resulting in the generated image being unnatural and without any semantics; and the optimization only at the initial noise stage cannot completely solve the problem of image-text mismatch, because attention may leak during the denoising stage. Summary of the invention

[0010] In view of the problems existing in the prior art, the purpose of the present invention is to provide a multi-target image generation method based on differentiated attention and denoising process guidance. The present invention solves the problem of semantic inconsistency between the generated image and the given text prompt in the current multi-target generation task in the field of text-generated images; the present invention does not require additional training or fine-tuning of the model, nor does it require additional control condition input during the image generation process. It directly controls the generation of each image through the constantly changing attention in an adaptive manner, which can effectively avoid the problems of target missing, semantic leakage, and attribute error during multi-target generation, and effectively improve the semantic matching between the generated image and the given text prompt.

[0011] The technical solution of the present invention is:

[0012] A multi-target image generation method based on differentiated attention and denoising process guidance, the steps of which include:

[0013] 1) Given a text prompt y consisting of N target words and an initial latent code z T , using the stable diffusion model to generate each target word in the text prompt y and the initial latent code z T Cross attention map CA T ;

[0014] 2) Through the loss function Optimize the initial latent code z used in the stable diffusion model T ;in,

[0015]

[0016] is the stable diffusion model for the initial latent code z T The cross-attention map corresponding to the i-th target word obtained after one inference, is the stable diffusion model for the initial latent code z T The cross-attention map corresponding to the j-th target word obtained after one inference, is the initial hidden code z T and the i-th target word y i The corresponding self-attention map, λ1, λ2 and λ3 are hyperparameters, and T is the total number of inferences;

[0017] 3) The stable diffusion model generates each target word in the text prompt y and the initial latent code z after optimization in step 2) T The optimal cross attention map between

[0018] 4) The stable diffusion model is used to optimize the initial latent code z after step 2) T Iterative denoising is performed and the cross-attention map generated at the set time step is interpolated; then the latent code z0 obtained after the iterative denoising is decoded to generate the image corresponding to the text prompt y.

[0019] Further, through CA t =αCA t +(1-α)CA T Interpolate the cross-attention map generated by the set iteration step; where T>t>σ, t is the current denoising time step, α is the scale weight of the interpolation, and σ is the time step used to limit the interpolation.

[0020] Furthermore, the stable diffusion model samples the initial latent code z from the standard Gaussian distribution N(0,I) T.

[0021] Furthermore, λ1, λ2 and λ3 are set to 1, 1 and 0.5 respectively, the interpolation scale weight α is set to 0.1, T=50, σ=35.

[0022] A server, characterized in that it comprises a memory and a processor, wherein the memory stores a computer program, the computer program is configured to be executed by the processor, and the computer program comprises instructions for executing the above method.

[0023] A computer-readable storage medium stores a computer program, wherein the computer program implements the above method when executed by a processor.

[0024] We propose a Divide-to-Conquer approach for generating high-fidelity images from multi-subject textual cues. We perform attention-based control in two divide-and-conquer phases, optimizing from the perspectives of noise initialization and denoising processes, respectively.

[0025] In the differentiation phase, the present invention optimizes the latent code by dividing the initial attention of different subjects into three parts: separation, activation and centralization. In the guidance phase, the present invention directly uses the optimized initial value to interpolate the forward attention at each time step, effectively controlling the denoising process.

[0026] A large number of experiments have demonstrated that the proposed method is superior to the original Stable Diffusion model and other existing state-of-the-art methods in generating semantically realistic images.

[0027] The advantages of the present invention are as follows:

[0028] The present invention can effectively improve the semantic consistency between the generated image and the given text prompt. Compared with the current attention-based method, the present invention fully considers the two stages of stable diffusion model reasoning: initial noise and denoising process, and adopts different strategies in the two stages: differentiation and guidance. The present invention updates the latent code through attention-level control in the initial noise stage, and uses the optimal attention map obtained from the optimized initial noise to guide the output of the diffusion model in the step-by-step denoising process. While solving the problem of inconsistent image-text matching in multi-target image generation, it avoids the potential out-of-distribution generation problem and improves the efficiency of image generation at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 It is a comparison diagram of the effects of the traditional stable diffusion model and the present invention.

[0030] Figure 2 The figure is a flow chart of the method of the present invention.

[0031] Figure 3 Cross-attention visualization for a failure case of the stable diffusion model.

[0032] Figure 4 Attention leakage diagram during denoising.

[0033] Figure 5 This is a qualitative result diagram.

[0034] Figure 6 Figure 2 is the ablation experiment result diagram of α and σ.

[0035] Figure 7 Generates effect diagram for complex text prompts.

[0036] Figure 8 Visualization of the optimized cross attention. DETAILED DESCRIPTION

[0037] The present invention is further described in detail below in conjunction with the accompanying drawings. The examples given are only used to explain the present invention but not to limit the scope of the present invention.

[0038] like Figure 2 As shown in the figure, the multi-target image generation method based on differentiated attention and denoising process guidance (Divide-to-Conquer, D2C) proposed in this invention is mainly divided into two stages: differentiation and guidance. In the differentiation stage, given the text prompt and random initial noise z T , the present invention respectively performs separation, activation and centering at the attention level to achieve optimal initialization; and in the guidance stage, the present invention replaces the attention map at the current denoising time step with the optimized initial attention map value interpolation at each time step, thereby ensuring the semantic alignment between the generated image and the text prompt.

[0039] 1. Background knowledge:

[0040] The stable diffusion model is a technique for synthesizing images using text, which includes an autoencoder and a denoiser. Specifically, given an image in pixel space The encoder ε(·) maps it to a hidden code z = ε(x). Then, the decoder Can be The input image is reconstructed. The denoiser in the stable diffusion is a text-conditioned UNet model that uses the pre-trained CLIP text encoder c in the latent space. θ (·) is used to train the text embedding obtained. Formally, given the latent code z of the image and the text hint y, the diffusion model ∈ θ The training is performed using squared error loss, and the loss formula is:

[0041]

[0042] Where ∈~N(0,I) represents Gaussian noise and t is the time step. During inference, the stable diffusion model samples the initial latent code z from the standard Gaussian distribution N(0,I) T , and denoised over time steps to generate z0. Then, z0 is passed to the decoder to obtain the final generated image.

[0043] The attention map can be formally calculated by the following formula: Where Q and K are the query and key respectively, and d is the channel dimension. In the stable diffusion model, there is cross attention (CA) between the words of the text and the image, while there is self-attention (SA) between the pixels of the image. For CA, the text prompt y is first encoded as a text embedding and then extracted as the key K. The latent code z of the image is also block-based and projected as the query Q, and finally CA is calculated according to the above formula. Similarly, in self-attention (SA), the latent code is used to calculate Q and K, and the calculation is performed in the same way. It should be noted that CA is the only interaction between the text prompt and the image, while SA shows the structure of the image, ensuring that the generated image is visually coherent and natural.

[0044] (II) Differentiation of initial attention:

[0045] To further study the attention mechanism, we first visualize the cross-attention graphs of some image-text mismatch cases generated by the stable diffusion model. Figure 3 As shown in Figure 2, the highlighted bright areas represent higher attention values, while the dark areas represent lower values. Studies have shown that the initial cross-attention map CA T (initial hidden code z T The cross attention between the given text condition y is crucial in the faithful generation of the image. Specifically, the overlap between the initial cross attention maps of different words, insufficient attention values, and scattered attention maps will eventually generate semantic mismatches in the generated image. Therefore, the present invention optimizes the initial noise by controlling the separation, activation, and centering of the cross attention.

[0046] Separation: Cross-attention reflects which region of the image each word in the text prompt focuses on. If multiple objects have high cross-attention values ​​for the same image region, their semantics will be expressed in this region, which will lead to semantic leakage. For example, Figure 3 As shown in the example on the left, the target word "dog" gives a high attention value to the overlapping area in the initial stage, and the target word "bird" also has a high attention value. Although they are in the final denoising time step ( and ), the attention map at the time of θ is corrected to a certain extent, but the attention overlap between “dog” and “bird” in the initial stage still leads to the final semantic leakage, causing the bird’s wings to overlap on the dog’s back.

[0047] To solve the overlapping problem, this paper aims to explicitly separate their cross-attention maps by minimizing the similarity between them. Given a text prompt y containing N targets, denoted by y sub ={y1,y2,…,y N}, the present invention minimizes the cosine similarity between the cross-attention maps of N targets through the following loss:

[0048]

[0049] in, is the cross-attention map corresponding to the i-th target word obtained after the stable diffusion model performs T inferences on the image x, where T is the number of inferences of the stable diffusion model, indicating that the present invention only performs this loss on the initial noise. By directly constraining the similarity between different cross-attentions, Preventing a region from being associated with multiple target words enables positional separation between targets in the output image.

[0050] Activation: In addition to indicating the corresponding attention region for each target word in the text prompt y, the cross-attention map also shows the degree of attention of each word in its region. The higher the cross-attention value, the higher the probability that the corresponding target word is generated in the corresponding region. Figure 3 As shown in the intermediate case, The attention value is significantly lower than The initial attention of the target “turtle” is insufficient. Therefore, the turtle fails to be generated in the image, resulting in the target missing problem.

[0051] Naturally, in order to improve the cross-attention value of the target word so as to obtain sufficient attention strength in the initial stage, the present invention adopts a method of directly activating and strengthening their cross-attention graphs, as shown below:

[0052]

[0053] By enhancing the maximum cross-attention of the least-attentive target, This ensures that the target is generated in the image.

[0054] Centralization: Through the above two parts, we can obtain cross-attention maps of different targets with high attention values ​​and non-overlapping areas. However, their cross-attentions may be dispersed in their respective areas. Figure 3 As shown in the example on the right, More concentrated, but As a result, in the generated image, the frog is not generated normally, but its flippers are generated on the bear's feet, indicating a semantic leakage problem.

[0055] To this end, the present invention introduces self-attention maps as a guide to focus cross attention, because the self-attention generated by the initial noise is proved to be more focused and structured. Specifically, for a target word y i , the present invention first selects the image patch with the highest cross attention:

[0056]

[0057] in Represents the two-dimensional coordinates of the image block. Then, the present invention minimizes the cosine distance between the cross attention map and the self-attention map by the following formula:

[0058]

[0059] in Represents the selected i-th target word y i Corresponding self-attention map. By enhancing the consistency of the cross-attention map and the self-attention map, the stable diffusion model enhances the target’s attention to the structure-related regions and further enhances the structure of the target.

[0060] Finally, the present invention integrates the three loss functions to obtain:

[0061]

[0062] where λ1, λ2, and λ3 are hyperparameters.

[0063] The present invention updates and optimizes the initial latent code z through loss T , and then the stable diffusion model generates the optimized initial latent code z T The optimal cross-attention map between each target word in the text prompt y To control the denoising process of the diffusion model and the generation of the image.

[0064] (III) Guiding the denoising process:

[0065] In the inference phase of the stable diffusion model, the text-conditioned UNet network iteratively denoises the latent code at each time step t (from T to 0). Since different inputs will get different attention maps, even if the optimal attention map is obtained in the initial stage, attention leakage may occur during the denoising process. Figure 4As shown in the figure, both the “monkey” and the “balloon” obtained the best attention in the initial stage, but during the denoising process, the attention of the “monkey” was seriously leaked, the area of ​​attention changed greatly, and the attention value was reduced. Finally, the monkey was not correctly generated in the image.

[0066] An intuitive solution is to iteratively optimize the hidden code at each time step. However, the optimized hidden code must be carefully calibrated at each time step during the denoising process to avoid out-of-distribution generation. Here, the present invention adopts a simple but effective attention-level interpolation operation to circumvent this challenge. The present invention uses the optimal cross-attention map obtained in the initial differentiation stage to The denoising process is conditioned by interpolating it at an early denoising stage:

[0067]

[0068] where t is the current denoising time step, α is the interpolation scale weight, and σ is used to limit the time step at which interpolation is performed. Note that the smaller α is, The greater the influence of , the more effectively the generation process can be controlled by optimizing the initial attention. Since this attention-level interpolation does not require back-propagation and iterative optimization of the latent code, it not only avoids potential out-of-distribution problems, but also improves generation efficiency.

[0069] All experiments were conducted on the open source Stable Diffusion model v1.4. The loss weights λ1, λ2, and λ3 were set to 1, 1, and 0.5, respectively. The interpolation scale weight α was set to 0.1, and the experiment was conducted in the first 15 of 50 inference time steps (T=50, σ=35). All experiments were conducted on a single NVIDIA A800 80G GPU.

[0070] Dataset: The present invention conducts experiments on the Attend-and-Excite dataset generated for multiple targets, which includes three subsets: animal-animal, animal-object, and object-object. Each text prompt is composed of two targets: "one [target 1] and one [target 2]". For the object target, a random color will be assigned to it, and then a text prompt is formed, for example: "one [animal] and one [color] [object]". The dataset includes 12 animals and 12 objects, so a total of 66 animal-animal text prompts and object-object text prompts, as well as 144 animal-object text prompts are obtained.

[0071] Evaluation Metrics: Given each text prompt, the present invention generates 64 images using 64 random seeds (the seeds remain consistent across all methods). In order to evaluate the alignment of text and image, the present invention uses CLIP image-text similarity and CLIP text-text similarity. Specifically, for CLIP image-text similarity, the present invention calculates the complete text prompt similarity by calculating the cosine similarity between the text prompt and the generated image in the CLIP feature space. The present invention further tests the performance on target missing by selecting the smaller value of the cosine similarity between the text prompt and the generated image of the two targets calculated, and calculates the similarity of the minimum target. For CLIP text-text similarity, the present invention uses BLIP to generate annotations for the generated image, and calculates the cosine similarity between the input text prompt and the generated annotation in the CLIP text feature space.

[0072] Experimental results: In order to verify the effectiveness of the present invention, it is compared with the native stable diffusion model Stable Diffusion and the most advanced methods including Composable Diffusion, Structure Diffusion, Attend-and-Excite, and InitNO.

[0073] The present invention Figure 5 Qualitative comparison results are provided in . In almost all cases, Composable Diffusion usually generates a fused target based on multiple given targets. Previous attention-based methods, such as StructureDiffusion and Attend-and-Excite, can synthesize multiple targets, but have the problem of semantic leakage. InitNO also generates unwanted blending and attribute binding errors due to attention leakage in the denoising stage. The present invention can solve the above problems and generate images that correctly match the given text prompts.

[0074] The present invention provides quantitative comparison results in Table 1. The Divide-to-Conquer method of the present invention consistently outperforms the existing methods in all three indicators of all subsets. Although the numerical improvement of the present method is relatively small, the superiority in text-image alignment and visual quality is well-founded, which is consistent with the qualitative results. In order to fully discuss the text fidelity of the generated images, the present invention further conducted a subjective user survey and reported the preference statistics in Table 1. The present invention observed that the vast majority of participants preferred the method of the present invention, indicating that the method of the present invention received better user feedback. In addition, the present invention provides a comparison of time efficiency in Table 2. In theory, the speed of stable diffusion model generation is closely related to the number of denoising steps. Since the attention map is calculated by each denoising step, each attention-based optimization of the hidden code will introduce additional denoising steps, resulting in low time efficiency. Therefore, the present invention compares the maximum number of total denoising steps with the previous attention-based method and provides a comparison result. From a practical analysis, the number of optimization iterations for different initial noises varies, so the present invention also reports the average generation time for each image in the entire dataset. These results demonstrate that our invention not only outperforms the state-of-the-art methods in generating images that are semantically aligned with given textual cues, but also excels in terms of the time efficiency of the synthesis.

[0075] Table 1 shows the quantitative comparison results.

[0076]

[0077]

[0078] Table 2 shows the time efficiency comparison

[0079] method Maximum number of denoising steps Average time Stable Diffusion 50 13.35s Attend-and-Excite 125 50.65s InitNO 175 33.42s Divide-to-Conquer 100 26.83s

[0080] Ablation experiment:

[0081] In order to verify the improvement effect of different modules of the present invention on algorithm performance, the present invention conducted a series of ablation analysis experiments.

[0082] The role of loss function. In the initial stage, the present invention conducts an ablation study on the three parts of the proposed loss function and reports the average CLIP scores of the three subsets in Table 3. The separation loss is used to segment the location of the target and the activation loss is used to support the generation of the target. Both loss functions can bring a large score improvement. The centralization loss provides a more focused attention and brings an improvement to text-to-text.

[0083] Attention interpolation. Table 3 also shows the decrease in CLIP score without interpolation, indicating that attention leakage during iterative denoising affects the alignment of text and image, and the best attention map in the initial stage may be suboptimal during denoising. In addition, the scale weight α of interpolation is also discussed. Figure 6 As shown in the upper part of , when α is greater than 0.6, the apple cannot be generated correctly. As α decreases, the semantic alignment of the images becomes better on both subjects. It is worth noting that when α is set to 0, the attention in the denoising process will be completely replaced by the initial attention. This is unacceptable because the noise levels at different time steps are different and the calculated attention should not be the same. This may lead to the loss of subject details, such as the bench and apple blending into the background (wall) in the generated image. The present invention sets α to 0.1, so that the attention map at each time step can be slightly fine-tuned. The present invention also Figure 6 The lower part of the paper conducts ablation experiments on the interpolation period hyperparameter σ. When σ is set to 50 (equal to T), the crown is not perfectly generated. As σ decreases, attention interpolation is applied to more time steps, resulting in a gradual improvement in the alignment of text and image. The present invention only applies interpolation in early time steps because the initial attention contains rough information. When σ is less than 30, the generated image lacks fine details and the image quality decreases. In order to strike a balance between text image alignment and image quality, the present invention sets σ to 35.

[0084] Figure 7 The generated images under complex text prompts are shown. When the text prompts contain more topics or more detailed descriptions, the traditional stable diffusion model becomes more uncontrollable, while the method of the present invention can still achieve semantic alignment between images and text prompts. In addition, Figure 8 Provided Figure 3 Visualization of the optimized cross-attention map of the case. Compared with the stable diffusion model, the method of the present invention separates the initial cross-attention map of the target, activates the attention value, and centralizes the high-value area. In addition, CA T and CA 0 The highly overlapping regions also indicate that the initial attention map effectively guides the denoising process and achieves better semantic alignment.

[0085] Table 3 Qualitative results of ablation experiments

[0086]

[0087] Although the specific embodiments of the present invention are disclosed for the purpose of illustration, the purpose is to help understand the content of the present invention and implement it accordingly, those skilled in the art will understand that various substitutions, changes and modifications are possible without departing from the spirit and scope of the present invention and the appended claims. Therefore, the present invention should not be limited to the content disclosed in the best embodiment, and the scope of the present invention is subject to the scope defined in the claims.

Claims

1. A multi-target image generation method based on differentiated attention and denoising process guidance, the steps comprising: 1) Given a text prompt y consisting of N target words and an initial latent code z T , using the stable diffusion model to generate each target word in the text prompt y and the initial latent code z T Cross attention map CA T ; 2) Through the loss function Optimize the initial latent code z used in the stable diffusion model T ;in, is the stable diffusion model for the initial latent code z T The cross-attention map corresponding to the i-th target word obtained after one inference, is the stable diffusion model for the initial latent code z T The cross-attention map corresponding to the j-th target word obtained after one inference, is the initial hidden code z T and the i-th target word y i The corresponding self-attention map, λ1, λ2 and λ3 are hyperparameters, and T is the total number of inferences; 3) The stable diffusion model generates each target word in the text prompt y and the initial latent code z after optimization in step 2) T The optimal cross attention map between 4) The stable diffusion model is used to optimize the initial latent code z after step 2) T Iterative denoising is performed and the cross-attention map generated at the set time step is interpolated; then the latent code z0 obtained after the iterative denoising is decoded to generate the image corresponding to the text prompt y.

2. The method according to claim 1, characterized in that: By CA t =αCA t +(1-α)CA T Interpolate the cross-attention map generated by the set iteration step; where T>t>σ, t is the current denoising time step, α is the scale weight of the interpolation, and σ is the time step used to limit the interpolation.

3. The method according to claim 2, characterized in that The stable diffusion model samples the initial latent code z from the standard Gaussian distribution N(0,I) T .

4. The method according to claim 1, 2 or 3, characterized in that: λ1, λ2 and λ3 are set to 1, 1 and 0.5 respectively, the interpolation scale weight α is set to 0.1, T = 50, σ = 35.

5. A server, characterized in that: The invention comprises a memory and a processor, wherein the memory stores a computer program, the computer program is configured to be executed by the processor, and the computer program comprises instructions for executing the method according to any one of claims 1 to 4.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.

Images