Multi-object image generation method based on differential attention and denoising process guidance

Abstract

The application discloses a multi-target image generation method based on differentiated attention and denoising process guidance, and the steps of the method comprise the following steps: 1) aiming at a given text prompt y comprising N target words and an initial hidden code z T , a stable diffusion model is used to generate a cross-attention graph CA T between each target word in the text prompt y and z T ; 2) the initial hidden code z T is optimized by setting a loss function; 3) the stable diffusion model generates an optimal cross-attention graph between each target word in the text prompt y and the optimized initial hidden code z T ; 4) the stable diffusion model performs iterative denoising on the optimized initial hidden code z T , and interpolates the cross-attention graphs generated at set time steps; and then the hidden code z0 obtained at the end of the iterative denoising is decoded to generate an image corresponding to the text prompt y. The application effectively improves the semantic matching between the generated image and the given text prompt.

Description

Technical Field

[0001] This invention belongs to the field of image generation technology and relates to a multi-target image generation method based on differentiation attention and denoising process guidance. Background Technology

[0002] The text-to-image generation task aims to generate visually natural and semantically faithful images based on text prompts. Traditional image generation methods involve designers drawing by hand or using software to create images based on text descriptions. In recent years, with the rapid development of artificial intelligence technology, various text-to-image generation techniques based on deep generative models have been widely applied in fields such as art design, digital development, and personalized customization. Compared to traditional methods, this image generation technology has achieved significant improvements in generation efficiency.

[0003] In the development of deep generative models, numerous technical branches have emerged, including variational autoencoders, generative adversarial networks, regularized flow, and diffusion models. Among them, diffusion models have been widely discussed and applied in both academia and industry due to their high-quality and diverse image generation capabilities. Within diffusion model methods, the stable diffusion model stands out as an open-source representative model for text-to-image generation, leading the trend of deep learning-based text-to-image generation technology.

[0004] However, even the most advanced stable diffusion models sometimes fail to generate images that perfectly match a given text cue, especially when generating multiple targets. Figure 1 As shown, this weakness mainly manifests in the following aspects: (1) target missing, that is, one or more targets in the text prompt cannot be generated normally; (2) semantic leakage, that is, one target leaks its semantics to another target, resulting in the mixing of multiple targets in the generated image; (3) attribute error, that is, the model incorrectly binds the target to its attributes. Therefore, how to achieve seamless alignment between the generated image and the given text prompt is crucial for the application of stable diffusion model technology.

[0005] Currently, in multi-objective image generation tasks, methods used to improve the consistency between images and text generated by stable diffusion models mainly fall into two categories: the first category provides additional control conditions for generation based on additional input, and the second category does not include additional input. Based on these two categories, the shortcomings of their respective background technologies are summarized as follows:

[0006] (1) Methods based on additional input

[0007] Methods based on additional input enhance the controllability of the generation process and improve the image-text matching ability of stable diffusion models by providing additional input information. Besides text prompts, additional image layout information (such as semantic segmentation maps or object bounding boxes) is typically introduced as input to constrain the object's position and size within the image. Recently, methods have also provided finer-grained control within the image space. Because these methods accept additional user input as control, the generated images are more likely to meet the user's generation requirements, and the image-text matching accuracy is higher. However, the input of these additional control conditions requires manual annotation, which is time-consuming and therefore impractical in many real-world applications.

[0008] (2) Method without additional input

[0009] In scenarios where only text prompts serve as control conditions, some methods attempt to generate multiple targets separately and then concatenate the model outputs to correctly output multiple targets in the generated image. Recently, increasing efforts have focused on attention-level control of generation. In stable diffusion models, since cross-attention reflects the region of text-interacting images and the intensity of the interaction, direct control at the cross-attention level can alter the expression intensity of each word in the text prompt, thereby improving text-image consistency. Guided by attention-level loss, some methods optimize the implicit coding at each time step of the diffusion model's denoising process, while others focus only on optimizing the initial noise. However, these methods are not comprehensive. Optimizing the implicit coding at each time step may lead to the generation of out-of-distribution images; that is, the implicit coding, after over-optimization, deviates from the model's original distribution manifold, resulting in unnatural and semantically meaningless generated images. Furthermore, optimization only at the initial noise stage cannot completely solve the image-text mismatch problem because attention may leak during the denoising stage. Summary of the Invention

[0010] To address the problems existing in the prior art, the present invention aims to provide a multi-object image generation method based on differentiation attention and a denoising process. This invention solves the problem of semantic inconsistency between the generated image and the given text prompt in current multi-object generation tasks in the field of text-to-image generation. This invention requires no additional model training or fine-tuning, nor any additional control conditions during image generation. It adaptively controls image generation through continuously and dynamically changing attention during the generation of each image, effectively avoiding problems such as target loss, semantic leakage, and attribute errors in multi-object generation, and significantly improving the semantic matching between the generated image and the given text prompt.

[0011] The technical solution of this invention is as follows:

[0012] A multi-target image generation method based on differentiation attention and denoising process guidance includes the following steps:

[0013] 1) Given a text prompt y containing N target words and an initial hidden code z) T The stable diffusion model is used to generate each target word in the text prompt y and the initial hidden code z. T Cross-attention graph (CA) between T ;

[0014] 2) Through the loss function The initial hidden code z used to optimize the stable diffusion model T ;in,

[0015]

[0016] It is a stable diffusion model for the initial hidden code z T The cross-attention map corresponding to the i-th target word obtained after one inference. It is a stable diffusion model for the initial hidden code z T The cross-attention map corresponding to the j-th target word obtained after one inference. It is the initial hidden code z T The i-th target word y i The corresponding self-attention map, λ1, λ2 and λ3 are hyperparameters, and T is the total number of inferences;

[0017] 3) The stable diffusion model generates each target word in the text prompt y and the initial hidden code z optimized in step 2). T Optimal cross-attention graph between and

[0018] 4) The stable diffusion model optimizes the initial hidden code z in step 2). T Perform iterative denoising and interpolate the cross-attention map generated at a set time step; then decode the hidden code z0 obtained after the iterative denoising to generate the image corresponding to the text prompt y.

[0019] Furthermore, through CA t =αCA t +(1-α)CA T Interpolate the cross-attention map generated by the set iteration step; where T>t>σ, t is the current denoising time step, α is the scale weight of the interpolation, and σ is the time step used to limit the interpolation.

[0020] Furthermore, the stable diffusion model samples the initial hidden code z from the standard Gaussian distribution N(0,I). T.

[0021] Furthermore, λ1, λ2 and λ3 are set to 1, 1 and 0.5 respectively, the interpolation scale weight α is set to 0.1, T = 50 and σ = 35.

[0022] A server is characterized by comprising a memory and a processor, the memory storing a computer program configured to be executed by the processor, the computer program including instructions for performing the methods described above.

[0023] A computer-readable storage medium having a computer program stored thereon, characterized in that the computer program implements the above-described method when executed by a processor.

[0024] This invention proposes a "divide-to-conquer" method for generating high-fidelity images from multi-subject text cues. The invention employs attention-based control through two divide-and-conquer phases, optimizing the noise initialization and denoising processes respectively.

[0025] During the differentiation phase, this invention optimizes the hidden coding by dividing the initial attention of different subjects into three parts: separation, activation, and centering. During the guidance phase, this invention directly interpolates the forward attention using the optimized initial values ​​at each time step, effectively controlling the denoising process.

[0026] Extensive experiments have demonstrated that this invention outperforms the original Stable Diffusion model and other state-of-the-art methods in generating semantically realistic images.

[0027] The advantages of this invention are as follows:

[0028] This invention effectively improves the semantic consistency between the generated image and the given text prompt. Compared with current attention-based methods, this invention fully considers the two stages of stable diffusion model inference: initial noise and denoising, and employs different strategies in each stage: differentiation and guidance. This invention updates the hidden code through attention-level control in the initial noise stage, and guides the output of the diffusion model using the optimal attention map obtained from the optimized initial noise during the progressive denoising process. This solves the problem of inconsistent image-text matching in multi-target image generation while avoiding potential out-of-distribution generation issues, and simultaneously improves the efficiency of image generation. Attached Figure Description

[0029] Figure 1 This is a comparison chart of the effects of the traditional stable diffusion model and the present invention.

[0030] Figure 2 This is a flowchart of the method of the present invention.

[0031] Figure 3 A visualization of cross-attention for cases of failure in a stable diffusion model.

[0032] Figure 4 This is an attention leak diagram during the noise reduction process.

[0033] Figure 5 This is a graph showing the qualitative results.

[0034] Figure 6 The figure shows the results of ablation experiments related to α and σ.

[0035] Figure 7 This is a screenshot showing the generated effect of complex text prompts.

[0036] Figure 8 This is the optimized visualization of cross-attention. Detailed Implementation

[0037] The present invention will now be described in further detail with reference to the accompanying drawings. The examples given are only for explaining the present invention and are not intended to limit the scope of the present invention.

[0038] like Figure 2 As shown, the proposed multi-objective image generation method (Divide-to-Conquer, D2C) based on differentiation attention and denoising process guidance mainly consists of two stages: differentiation and guidance. In the differentiation stage, given text prompts and random initial noise z... T This invention performs separation, activation, and centralization at the attention level to achieve optimal initialization; and in the guidance phase, this invention replaces the attention map at the current denoising time step with the optimized initial attention map value at each time step, thereby ensuring semantic alignment between the generated image and the text prompt.

[0039] (I) Background Knowledge:

[0040] The stable diffusion model is a technique for synthesizing images using text, comprising an autoencoder and a denoiser. Specifically, given an image within a pixel space... The encoder ε(·) maps it to the hidden code z = ε(x). Then, the decoder... It can be done The input image is reconstructed. The denoiser in the stable diffusion is a text-conditional UNet model that uses a pre-trained CLIP text encoder in the latent space. θ (·) The obtained text embeddings are used for training. Formally, given the latent code z of an image and the text cue y, the diffusion model ∈ θ Training is performed using the squared error loss formula, which is:

[0041]

[0042] Where ∈ ~ N(0,I) represents Gaussian noise, and t is the time step. During inference, the stable diffusion model samples the initial hidden code z from the standard Gaussian distribution N(0,I). T The noise is gradually reduced over time steps to generate z0. Then, z0 is passed to the decoder. To obtain the final generated image.

[0043] Attention maps can be formally calculated using the following formula: Here, Q and K are the query and key, respectively, and d is the channel dimension. In the stable diffusion model, there is cross-attention (CA) between the words of the text and the image, while there is self-attention (SA) between the pixels of the image. For CA, the text cue y is first encoded as a text embedding and then extracted as the key K. The latent code z of the image is also segmented and projected as the query Q, and finally CA is calculated according to the above formula. Similarly, in self-attention (SA), the latent code is used to calculate Q and K, and the calculation is performed in the same way. It is important to note that CA is the only interaction between the text cue and the image, while SA reveals the structure of the image, ensuring that the generated image is visually coherent and natural.

[0044] (II) Differentiation of initial attention:

[0045] By delving into the attention mechanism, this invention first visualizes the cross-attention maps of text-image mismatch cases generated by a stable diffusion model. For example... Figure 3 As shown, the highlighted bright areas represent higher attention values, while the dark areas represent lower values. Research indicates that the initial cross-attention map (CA)... T (initial hidden code z) T Cross-attention (between given text condition y) is crucial for faithful image generation. Specifically, overlap between initial cross-attention maps of different words, insufficient attention values, and scattered attention maps all contribute to semantic mismatches in the generated images. Therefore, this invention optimizes the initial noise by controlling the separation, activation, and centering of cross-attention.

[0046] Separation: Cross-attention reflects which region of the image each word in the text prompt focuses on. If multiple targets have high cross-attention values ​​for the same image region, then their semantics will all be expressed within that region, leading to semantic leakage. For example, ... Figure 3 As shown in the example on the left, the target word "dog" initially received a high attention value for the overlapping region, while the target word "bird" also received a high attention value. Although they received a high attention value in the final denoising time step ( and The attention map was corrected to some extent, but the overlap of attention between "dog" and "bird" in the initial stage still led to the final semantic leakage, causing the bird's wings to overlap on the dog's back.

[0047] To address the overlap problem, this invention aims to explicitly separate cross-attention maps by minimizing their similarity. Given a text cue y containing N targets, denoted as yi... sub ={y1,y2,…,y N This invention minimizes the cosine similarity between the cross-attention maps of N targets using the following loss:

[0048]

[0049] in, This is the cross-attention map corresponding to the i-th target word obtained after the stable diffusion model performs T inferences on image x, where T is the number of inferences in the stable diffusion model, indicating that this loss is only applied to the initial noise. By directly constraining the similarity between different cross-attentions, This prevents a region from being associated with multiple target words, thereby achieving positional separation between targets in the output image.

[0050] Activation: In addition to indicating the corresponding attention region for each target word in the text prompt y, the cross-attention map also shows the degree of attention each word receives within its region. A higher cross-attention value indicates a higher probability of the target word being generated in the corresponding region. For example... Figure 3 As shown in the intermediate case. Attention scores were significantly lower than The initial attention to the target "turtle" was insufficient. Therefore, turtle generation failed in the image, resulting in a missing target problem.

[0051] Naturally, in order to improve the cross-attention value of the target words and obtain sufficient attention intensity in the initial stage, the present invention adopts a method of directly activating and strengthening their cross-attention maps, as shown below:

[0052]

[0053] By enhancing the maximum cross-attention of the target with the lowest attention level. This ensures that the target is generated in the image.

[0054] Centralization: Through the above two parts, we can obtain cross-attention maps of different targets with high attention values ​​and non-overlapping regions. However, their cross-attention may be scattered within their respective regions. For example... Figure 3 As shown in the case on the right, It is relatively concentrated, but The images are somewhat scattered. This results in the frog not being generated correctly, but instead its webbed feet being generated on the bear's feet, indicating a semantic leakage issue.

[0055] To this end, this invention introduces self-attention maps as guidance to focus cross-attention, since self-attention generated by initial noise has proven to be more focused and structure-preserving. Specifically, for a target word y i The present invention first selects the image patch with the highest cross-attention:

[0056]

[0057] in The two-dimensional coordinates of the image patch are represented. Then, this invention minimizes the cosine distance between the cross-attention map and the self-attention map using the following formula:

[0058]

[0059] in This indicates the relationship with the selected i-th target word y. i The corresponding self-attention map. By enhancing the consistency between the cross-attention map and the self-attention map, the stable diffusion model strengthens the target's focus on structure-related regions, further enhancing the target's structure.

[0060] Finally, this invention integrates the three loss functions to obtain:

[0061]

[0062] λ1, λ2, and λ3 are hyperparameters.

[0063] This invention updates and optimizes the initial hidden code z using a loss function. T Then, the stable diffusion model generates the optimized initial hidden code z. T Optimal cross-attention graph between the target word and the text prompt y This is used to control the denoising process of the diffusion model and the generation of images.

[0064] (III) Guided noise reduction process:

[0065] During the inference phase of the stable diffusion model, the text-conditional UNet network iteratively denoises the hidden code at each time step t (from T to 0). Since different inputs yield different attention maps, even if the optimal attention map is obtained initially, attention leakage may occur during the denoising process. Figure 4As shown, both the "monkey" and the "balloon" achieved optimal attention in the initial stage. However, during the denoising process, the "monkey's" attention was severely leaked, the area of ​​focus changed significantly, and the attention value decreased. Ultimately, the monkey was not able to be correctly generated in the image.

[0066] An intuitive solution is to iteratively optimize the hidden code at each time step. However, optimizing the hidden code at each time step during denoising requires careful calibration to avoid out-of-distribution generation. This invention circumvents this challenge by employing a simple yet effective attention-level interpolation operation. This invention utilizes the optimal cross-attention map obtained in the initial differentiation phase. The denoising process is modulated by interpolating it in the early stages of denoising:

[0067]

[0068] Where t is the current denoising time step, α is the interpolation scale weight, and σ is used to limit the time step for interpolation. Note that the smaller α is, the better. The greater the influence, the more effectively the generation process can be controlled by optimizing the initial attention. Since this attention-level interpolation does not require backpropagation and iterative optimization of the underlying code, it not only avoids potential out-of-distribution problems but also improves generation efficiency.

[0069] All experiments in this invention were conducted on the open-source Stable Diffusion v1.4 model. The loss weights λ1, λ2, and λ3 were set to 1, 1, and 0.5, respectively. The interpolation scale weight α was set to 0.1, and experiments were performed in the first 15 of 50 inference time steps (T = 50, σ = 35). All experiments were performed on a single NVIDIA A800 80G GPU.

[0070] Dataset: This invention was experimented on the Attend-and-Excite dataset, which generates text cues for multiple targets. This dataset comprises three subsets: animal-animal, animal-object, and object-object. Each text cue is composed of two targets: "a [target 1] and a [target 2]". For object targets, a random color is assigned, and the text cue is then constructed, for example: "a [animal] and a [color] [object]". The dataset includes 12 animals and 12 objects, resulting in a total of 66 animal-animal text cues, 144 object-object text cues, and 144 animal-object text cues.

[0071] Evaluation Metrics: Given each text prompt, this invention generates 64 images using 64 random seeds (seeds remain consistent across all methods). To evaluate the alignment between text and images, this invention employs CLIP image-text similarity and CLIP text-text similarity. Specifically, for CLIP image-text similarity, this invention calculates the full text prompt similarity by computing the cosine similarity between the text prompt and the generated image in the CLIP feature space. This invention further tests performance for missing targets by selecting the smaller value of the calculated cosine similarity between the text prompts and generated images of two targets, and calculates the minimum similarity of the target. For CLIP text-text similarity, this invention uses BLIP to generate annotations for the generated images and calculates the cosine similarity between the input text prompt and the generated annotations in the CLIP text feature space.

[0072] Experimental Results: To verify the effectiveness of this invention, it was compared with the native stable diffusion model Stable Diffusion and state-of-the-art methods including Composable Diffusion, Structure Diffusion, Attend-and-Excite, and InitNO.

[0073] This invention is in Figure 5 The results provide qualitative comparisons. In almost all cases, Composable Diffusion typically generates a fused target based on multiple given targets. Previous attention-based methods, such as StructureDiffusion and Attend-and-Excite, can synthesize multiple targets, but suffer from semantic leakage. InitNO also generates unwanted blending and attribute binding errors due to attention leakage in the denoising stage. This invention solves the above problems and generates an image that correctly matches the given text prompt.

[0074] The present invention provides quantitative comparison results in Table 1. The Divide-to-Conquer method of the present invention consistently outperforms existing methods across all three metrics in all subsets. Although the numerical improvement of the present invention's method is relatively small, its superiority in text-image alignment and visual quality is well-founded, consistent with the qualitative results. To comprehensively discuss the text fidelity of the generated images, the present invention further conducted a subjective user survey, and the statistical data on preferences are reported in Table 1. The present invention observed that the vast majority of participants preferred the present invention's method, indicating that the present invention's method received better user feedback. Furthermore, the present invention provides comparison results on time efficiency in Table 2. Theoretically, the generation speed of a stable diffusion model is closely related to the number of denoising steps. Since the attention map is computed through each denoising step, each attention-based optimization of the implicit coding introduces an additional denoising step, leading to inefficiency. Therefore, the present invention compares the maximum number of total denoising steps with previous attention-based methods and provides the comparison results. In practice, the number of optimization iterations varies with different initial noise levels; therefore, the present invention also reports the average generation time per image across the entire dataset. These results demonstrate that the present invention not only surpasses state-of-the-art methods in semantic alignment of generated images with given text prompts, but also excels in terms of time efficiency in synthesis.

[0075] Table 1 shows the quantitative comparison results.

[0076]

[0077]

[0078] Table 2 shows a comparison of time efficiency.

[0079] method Maximum number of noise reduction steps Average time Stable Diffusion 50 13.35s Attend-and-Excite 125 50.65s InitNO 175 33.42s Divide-to-Conquer 100 26.83s

[0080] Ablation experiment:

[0081] To verify the performance improvement effect of different modules of the present invention on the algorithm, a series of ablation analysis experiments were conducted.

[0082] The role of the loss function. In the initial stage, this invention conducted an ablation study on the three components of the proposed loss function and reported the average CLIP scores of the three subsets in Table 3. The separation loss is used to segment the location of the target, while the activation loss is used to support the generation of the target; both loss functions can significantly improve the score. The centering loss provides more focused attention, resulting in improved text-to-text performance.

[0083] Attention interpolation. Table 3 also shows the decrease in CLIP score without interpolation, indicating that attention leakage during iterative denoising affects text-image alignment, and the optimal attention map in the initial stage may be suboptimal during denoising. Furthermore, this invention discusses the scale weight α of interpolation. Figure 6 As shown in the upper part, when α is greater than 0.6, the apple cannot be generated correctly. As α decreases, the semantic alignment of the image improves on both subjects. It is noteworthy that when α is set to 0, the attention during the denoising process is completely replaced by the initial attention. This is unacceptable because the noise level is different at different time steps, and the calculated attention should not be the same. This can lead to the loss of subject details, such as the bench and apple blending into the background (wall) in the generated image. This invention sets α to 0.1, allowing for slight fine-tuning of the attention map at each time step. This invention also... Figure 6 The lower part of the paper presents an ablation experiment on the interpolation period hyperparameter σ. When σ is set to 50 (equal to T), the crown is not perfectly generated. As σ decreases, attention interpolation is applied to more time steps, resulting in a gradual improvement in the alignment between the text and the image. This invention applies interpolation only in early time steps because the initial attention contains coarse information. When σ is less than 30, the generated image lacks fine details, and the image quality deteriorates. To achieve a balance between text-image alignment and image quality, this invention sets σ to 35.

[0084] Figure 7 The image generated under complex text prompts is shown. When the text prompts contain more topics or more detailed descriptions, traditional stable diffusion models become more uncontrollable, while the method of this invention can still achieve semantic alignment between the image and the text prompts. Furthermore, Figure 8 Provided Figure 3 Visualization of the optimized cross-attention map of the case. Compared to the stable diffusion model, the method of this invention separates the initial cross-attention map of the target, activates attention values, and centers high-value regions. Furthermore, CA... T and CA 0 The highly overlapping regions also indicate that the initial attention map efficiently guided the denoising process and achieved better semantic alignment.

[0085] Table 3 Qualitative results of ablation experiments

[0086]

[0087] Although specific embodiments of the invention have been disclosed for illustrative purposes to aid in understanding and implementing the invention, those skilled in the art will understand that various substitutions, variations, and modifications are possible without departing from the spirit and scope of the invention and the appended claims. Therefore, the invention should not be limited to the content disclosed in the preferred embodiments, and the scope of protection claimed by the invention is defined by the claims.

Claims

1. A multi-target image generation method based on differentiation attention and denoising process guidance, comprising the following steps: 1) Given a text prompt y containing N target words and an initial hidden code z) T The stable diffusion model is used to generate each target word in the text prompt y and the initial hidden code z. T Cross-attention graph (CA) between T ; 2) Through the loss function The initial hidden code z used to optimize the stable diffusion model T ;in, It is a stable diffusion model for the initial hidden code z T The cross-attention map corresponding to the i-th target word obtained after one inference. It is a stable diffusion model for the initial hidden code z T The cross-attention map corresponding to the j-th target word obtained after one inference. It is the initial hidden code z T The i-th target word y i The corresponding self-attention map, λ1, λ2 and λ3 are hyperparameters, and T is the total number of inferences; 3) The stable diffusion model generates each target word in the text prompt y and the initial hidden code z optimized in step 2). T Optimal cross-attention graph between 4) The stable diffusion model optimizes the initial hidden code z in step 2). T Perform iterative denoising and interpolate the cross-attention map generated at a set time step; then decode the hidden code z0 obtained after the iterative denoising to generate the image corresponding to the text prompt y.

2. The method according to claim 1, characterized in that, Through CA t =αCA t +(1-α)CA T Interpolate the cross-attention map generated by the set iteration step; where T>t>σ, t is the current denoising time step, α is the scale weight of the interpolation, and σ is the time step used to limit the interpolation.

3. The method according to claim 2, characterized in that, The stable diffusion model samples the initial hidden code z from a standard Gaussian distribution N(0,I). T .

4. The method according to claim 1, 2, or 3, characterized in that, λ1, λ2 and λ3 are set to 1, 1 and 0.5 respectively, the interpolation scale weight α is set to 0.1, T = 50 and σ = 35.

5. A server, characterized in that, It includes a memory and a processor, the memory storing a computer program configured to be executed by the processor, the computer program including instructions for performing the method of any one of claims 1 to 4.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 4.

Images