Text generation image method based on attention modulation and text paraphrasing

Abstract

The application provides a controllable text generation image method based on attention modulation and text restatement. The method can generate a high-definition image with specific layout and semantic consistency according to a text description. Specifically, in the first stage, the image layout information is generated according to the input text content through the visual concept understanding and layout arrangement capability of a large language model. In the second stage, the attention map in the diffusion process is adaptively adjusted by using a layout-based attention modulation mechanism to ensure that the objects are generated within the given layout. In the third stage, the image translation model is used to regenerate the text, and a semantic regeneration supervision strategy is designed to constrain the regenerated text to be semantically aligned with the input text, thereby indirectly correcting the error generation area. A large amount of experimental data and qualitative results show that the application performs well in the text generation image task, significantly reduces the operation complexity in a training-free manner, and does not require the user to give specific layout information, thereby facilitating the use of the user.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of image generation and computer vision, and particularly relates to a text-to-image generation method based on attention modulation and text restatement, aiming to improve the semantic alignment capability and diversified structure expression of text-to-image generation. BACKGROUND

[0002] In recent years, large-scale diffusion models have been used as the main tool for image generation to synthesize diversified and high-quality images, because their performance is superior to that of generative adversarial networks and autoregressive models, and they are stable to train. Since 2020, major breakthroughs have been made in large-scale diffusion models, especially the emergence of stable diffusion models, which have greatly promoted the development of high-quality and semantically consistent text-to-image synthesis. However, text descriptions can only provide coarse-grained color and style information and cannot give specific layouts of images. Therefore, the research on controllable text-to-image generation algorithms based on layouts is a promising direction.

[0003] Some methods focus on improving the controllability of text-to-image generation based on layouts, hoping that objects in the text will be generated within specific layouts. The main practice is to inject layout information into pre-trained stable diffusion models as additional conditions to guide the image generation process. GLIGEN proposes a learnable gated self-attention layer to integrate layout information into stable diffusion models, achieving controllable text-to-image generation in open domains. BoxDiff proposes a box-constrained diffusion model and designs box-in-constraint, box-out-constraint, and corner-constraint to ensure that objects are generated in specified areas. These two classic methods provide innovative ideas and correct guidance for subsequent research. However, these methods are indirect guidance methods that do not directly adjust the attention content, often resulting in small object loss and inaccurate generation location problems. Based on this, the effective attention modulation mechanism we propose can directly and adaptively adjust the attention map during the generation process, effectively alleviating the small object loss problem.

[0004] Recently, large language models (LLMs) have shown excellent performance in cross-modal tasks due to their strong contextual learning and layout reasoning capabilities. Therefore, we use a large language model as a layout generator and design text prompts to guide the large language model to return specific object labels and bounding box information. However, large language models can only provide layout content and cannot effectively combine the high-level spatial relationships of multiple object layout information. Based on this, we propose a text restatement approach that uses the differences between the restated text and the input text to update the hidden space vectors during the diffusion process, and through multiple iterations, corrects the incorrect generation content to alleviate the inaccurate generation problem.

[0005] In summary, current text-to-image generation methods can achieve basic generation tasks, but these methods require large datasets containing millions of images and powerful computational resources, making them very expensive to implement. Furthermore, given a specific layout, they cannot accurately generate images consistent with the text prompts, and they suffer from issues such as attribute coupling, illogical spatial relationships, and missing objects. Although some recent methods have made progress through additional training, these indirect approaches cannot fundamentally solve the aforementioned problems. In conclusion, current generative models cannot plan the image layout based on text content, nor can they accurately generate high-quality, diverse images based on the layout conditions. Summary of the Invention

[0006] To address the above problems, this invention proposes a text-to-image generation method based on attention modulation and text restatement, comprising the following steps:

[0007] 1. A text-to-image generation method based on attention modulation and text restatement, which can adaptively adjust the attention map during the diffusion process and automatically generate image layouts using a large language model, lowering the user threshold; and corrects incorrect generated regions through text restatement, including the following steps:

[0008] S1. Construct a large language model prompting framework, which guides the large language model to automatically generate object labels and layout position information through text prompts; the text content is encoded into the latent space through a pre-trained text encoder as a guide for the diffusion process during generation.

[0009] S2, Construct a stable diffusion model for the latent space, which can receive text encoding and layout information and compress them into a low-dimensional space;

[0010] S3, construct an attention modulation module to modulate the cross-attention map in the diffusion process of S2 to enhance the content generation of the region of interest and alleviate the problem of small object loss;

[0011] S4. Construct a semantic text regeneration supervision network. Restate the image in S2 after adjustment in S3 using a large image translation model and perform semantic supervision with the input text to update the latent features of each iteration step in S2.

[0012] S5 uses a pre-trained image decoder to recover an image from the hidden features obtained in the last step of S2 after iteration.

[0013] 2. Step S1 includes the following steps:

[0014] S11, Design the prompt word framework, which includes instructions that describe the task and specify the response format;

[0015] S12, input the text content entered by the user into the prompt word frame in S11, and use the large language model to generate object labels and layout information, which is represented as:

[0016] (1)

[0017] 3. Step S2 specifically includes:

[0018] S21, First, a pre-trained text encoder is used to encode the text input into a text vector to guide the generation diffusion process; this part is represented as follows:

[0019] (2)

[0020] S22, initialize low-dimensional random noise with a dimension of 64×64, which satisfies a standard Gaussian distribution, and use it as the starting point of the diffusion process;

[0021] S23, map the label information obtained in S12 onto the image with the same noise dimension, and store it in the database. This part is represented as:

[0022] (3)

[0023] S24: Construct a pre-trained stable diffusion model, receiving the text vector encoded in S21; starting with the initial random pure Gaussian noise from S22, begin the diffusion process, gradually denoising. This part is represented as:

[0024] (4)

[0025] in , This represents the total number of iterations. Indicates the first Noise characteristics of the step, It is a text vector; For noise prediction networks, It is related to the choice of the mean coefficient;

[0026] 4. Step S3 specifically includes:

[0027] S31, the first step in capturing the diffusion process Input features of the cross-attention layer at the next iteration step Construct three linear layers to obtain the first... In the second loop Cross-attention input for each iteration step: query ,key ,value This part is represented as:

[0028] (5)

[0029] (6)

[0030] (7)

[0031] In the formula , , These represent the weights of three different linear mapping layers; The number of iterations. This indicates the number of iterations in the loop, which is related to the convergence of the text restatement;

[0032] S32, Obtain the cross-attention map This part is expressed as:

[0033] (8)

[0034] In the formula This represents the activation function. This indicates the transpose operation. express and Dimension, used for Perform normalization processing;

[0035] S33, Construct an attention modulation module, which adaptively modulates the cross-attention map obtained in S32 based on the layout information obtained in S23. This part is expressed as:

[0036] (9)

[0037] (10)

[0038] In the formula The mask representing the bounding box. This indicates the transpose operation. Indicates the first The proportion of the bounding box region to the entire image;

[0039] S34, use an image decoder to decode the latent features at iteration step 0, recovering the pixel space from the low-dimensional latent space to obtain the restatement iterations. A high-resolution image, this part is expressed as:

[0040] (11)

[0041] 5. Step S4 specifically includes:

[0042] S41, Construct a large-scale image translation model, for the first... The generated image of the wheel is restated with text, and this part is expressed as:

[0043] (11)

[0044] S42, using the same text encoder as S21, performs the following steps: The content of the round text restatement is encoded, and this part is expressed as:

[0045] (12)

[0046] S43, high-level semantic supervision is performed on the restated text vector and the initial text vector to indirectly and implicitly correct incorrect semantically generated content. This part is expressed as:

[0047] (13)

[0048] In the formula This indicates the calculation of cosine similarity. Represents logarithmic operations. This indicates a restatement of the loss;

[0049] S44, utilizing high-level semantic supervision loss, for the first... The diffusion process of the wheel The latent features of the iteration step are updated, and a new round of diffusion is executed; this part is expressed as:

[0050] (14)

[0051] Beneficial Effects: Compared with existing technologies, this invention provides a text-to-image generation method based on attention modulation and text restatement, which can accurately generate high-quality images with semantic consistency and close alignment with the image layout according to the text and image layout, and produces the following beneficial effects:

[0052] 1. Lowering the user barrier: Existing technologies often require users to provide fine-grained layout information when generating images based on layout information, which is very difficult for users. This invention avoids an excessively high user barrier by designing prompt templates and using a large language model to automatically generate layout information based on text content.

[0053] 2. No Training Required: Traditional training methods require a large amount of paired text-image data and significant computational resources to train a diffusion model with a large number of parameters. The difficulty in obtaining paired text-image data, the high cost of computational resources, and the long training time hinder the practical deployment of the model. This method designs a training-free architecture, utilizing the generative capabilities of a large base model to construct an external supervision function to update the latent features in the stable diffusion process. This method significantly reduces memory requirements and resource consumption, enabling the generation of an image within 7 seconds, demonstrating excellent real-time generation capabilities.

[0054] 3. User-Adaptive Adjustment: While existing methods can capture defects in generated images, they lack effective ways to correct erroneous content. In contrast, the text restatement stage proposed in this method effectively alleviates this deficiency. Text restatement is particularly user-friendly; by designing the number of iterations, it continuously corrects erroneous areas and adaptively adjusts unreasonable generated content, ultimately producing the desired result for the user. Attached Figure Description

[0055] Figure 1 This is a flowchart illustrating the text-to-image generation process of this invention.

[0056] Figure 2 This invention demonstrates the text generation effect of a multi-object object containing multiple attributes.

[0057] Figure 3 This is a visualization comparison of the results of this invention with other methods.

[0058] Figure 4 This is a visualization of the ablation experiment results for each of the main parts of this invention.

[0059] Figure 5 This is a visualization of the attention map results before and after restating the iteration rounds and attention modulation in this invention.

[0060] Figure 6 and Figure 7 The figure shows the quantitative comparison results of the controllable text generation image method based on attention modulation and text restatement of the present invention with other methods on four datasets: HRS, CC-500, NSR-1K and DrawBench.

[0061] Figure 8 This is a comparison chart showing the generation effects of different large language models used in this invention.

[0062] Figure 9 and Figure 10 These are the prompt template and several layout generation result diagrams of this invention. Detailed Implementation

[0063] The controllable text-to-image generation method based on attention modulation and text restatement of this invention utilizes a stable diffusion model and a large language model, enabling the model to automatically generate layout information based on the text and effectively combine text content and layout information to generate high-quality images. Simultaneously, the training-free approach significantly reduces computational costs, greatly enhancing the practical application value of this method.

[0064] The invention will be further described below with reference to specific embodiments.

[0065] Example 1:

[0066] like Figure 1 As shown, the image generation process is as follows: User input text: a yellow dog and a blue chair. First, the user input text is used to generate image layout and main object labels through a large language model. Then, the layout and label information are mapped to a layout representation with the same dimension as random noise. Next, a pre-trained stable diffusion model is loaded and a low-dimensional random noise is initialized. The input text is mapped to a text vector using a pre-trained text encoder, and this text vector and random noise are simultaneously input into the pre-trained stable diffusion model for diffusion generation. An attention modulation module is constructed to extract the cross-attention map during the diffusion process. The attention map and the same-dimensional layout representation are input into the attention modulation module for layout-based modulation. After all iterations are completed, the stable diffusion model outputs an image. Finally, a semantic text regeneration module is constructed to translate the generated image into restated text using an image translation model, and then encodes it into a restated text vector using a text encoder. This restated text vector is supervised against the input text vector, and the loss is backpropagated to the latent features through gradient penalty to update the latent features. This process continues until the loop stops.

[0067] The specific process of the attention modulation mechanism is as follows: Image layout is decoupled based on object labels, ensuring a single label corresponds to its corresponding object layout. The ratio of the area of ​​the object layout corresponding to each label to the total image area is calculated. Subsequently, the cross-attention map is adaptively adjusted based on this area ratio. Specifically, to emphasize the main objects in the text, we need to adjust the attention map to focus more on aspects of interest.

[0068] The specific process of text restatement is as follows: After setting the number of iterations, attention modulation is applied during the diffusion process in each round, and the iteration step tends to 0 to generate the image for that round. A pre-trained image translation model translates this image into restated text, which is then encoded into a text vector using a text encoder. The loss between the restated text vector and the input text vector in the high-dimensional semantic space is calculated. This loss is used to update the latent features during the diffusion process through gradient backpropagation, and the process continues into the next round of iteration. Iteration stops when the loop ends or the loss is less than 0.01, at which point the generated image is input again.

[0069] This invention provides a controllable text-to-image generation method based on attention modulation and text restatement. This method enhances the alignment of the synthesized image with the given text and corrects erroneous regions promptly through restatement. This approach enables the generative model to handle complex text descriptions, avoiding issues such as attribute coupling, unreasonable spatial relationship expressions, and missing objects. Furthermore, by automatically generating image layouts based on the text using a large language model, complex layout information is not required from the user, significantly lowering the barrier to entry for users.

[0070] Figure 2 This paper presents the effect of the controllable text-to-image generation method based on attention modulation and text restatement of the present invention on text containing multiple objects with multiple attributes. It can be seen that this method can accurately generate images that match the text and layout when faced with multiple attributes, multiple objects, and complex spatial relationships.

[0071] Figure 3 This is a visualization comparison of the controllable text-to-image generation method based on attention modulation and text restatement of the present invention with other methods. It can be observed that the controllable text-to-image generation method based on attention modulation and text restatement produces higher quality images with stronger semantic alignment than other methods.

[0072] Figure 4 The images show the ablation experiment visualization results of the main components of the controllable text generation image method based on attention modulation and text restatement proposed in this invention. It can be seen that the attention modulation module and text restatement module are crucial to this method.

[0073] Figure 5 This image shows the attention map visualization results of the controllable text-to-image generation method based on attention modulation and text restatement of the present invention before and after the number of restatement iterations and attention modulation. It can be seen that the more restatement iterations, the better the correction effect on erroneous generated regions, and the higher the alignment between the image and text. Simultaneously, the region of interest increases after attention modulation, resulting in better performance.

[0074] Figure 6 and Figure 7This figure shows the quantitative comparison results of the controllable text generation image method based on attention modulation and text restatement of the present invention with other methods on four datasets: HRS, CC-500, NSR-1K, and DrawBench. It can be seen that the controllable text generation image method based on attention modulation and text restatement performs better than the other methods.

[0075] Figure 8 The diagram shows a comparison of the generation performance of the controllable text-to-image generation method based on attention modulation and text restatement of this invention using different large language models. It can be observed that the more continuously updated the large language model, the stronger its visual planning capability, which improves the performance of this method.

[0076] Figure 9 and Figure 10 These are, respectively, the prompt templates and several layout generation result images of the controllable text-to-image method based on attention modulation and text restatement of this invention. It can be seen that, through a carefully designed prompt template, the visual understanding and planning capabilities of the large language model can be fully mobilized, resulting in a reasonable image layout.

[0077] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

[0078] While the specific embodiments of the present invention have been described above, they are not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A text-to-image generation method based on attention modulation and text restatement, characterized in that... It can adaptively adjust the attention map during the diffusion process and automatically generate image layouts using a large language model, lowering the user threshold; and it corrects incorrect generated regions through text restatement, including the following steps: S1. Construct a large language model prompting framework, which guides the large language model to automatically generate object labels and layout position information through text prompts; the text content is encoded into the latent space through a pre-trained text encoder as a guide for the diffusion process during generation. S2, construct a stable diffusion model for the latent space, which can receive text encoding and layout information and compress them into a low-dimensional space; step S2 specifically involves: S21, firstly, a pre-trained text encoder is used to encode the text input into a text vector, which is used to guide the generation diffusion process. This part is represented as: (1) S22, initialize low-dimensional random noise with a dimension of 64×64, which satisfies a standard Gaussian distribution, and use it as the starting point of the diffusion process; S23, map the label information obtained in S12 onto the image with the same noise dimension, and store it in the database. This part is represented as: (2) S24: Construct a pre-trained stable diffusion model, receiving the text vector encoded in S21; starting with the initial random pure Gaussian noise from S22, begin the diffusion process, gradually denoising. This part is represented as: (3) in , This represents the total number of iterations. Indicates the first Noise characteristics of the step, It is a text vector; For noise prediction networks, It is related to the choice of the mean coefficient; S3, construct an attention modulation module to modulate the cross-attention map in the diffusion process of S2 to enhance the content generation of the region of interest and alleviate the problem of small object loss; the specific steps of S3 are as follows: S31, the first step in capturing the diffusion process Input features of the cross-attention layer at the next iteration step Construct three linear layers to obtain the first... In the second loop Cross-attention input for each iteration step: query ,key ,value This part is represented as: (4) (5) (6) In the formula , , These represent the weights of three different linear mapping layers; The number of iterations. This indicates the number of iterations in the loop, which is related to the convergence of the text restatement; S32, Obtain the cross-attention map This part is expressed as: (7) In the formula This represents the activation function. This indicates the transpose operation. express and Dimension, used for Perform normalization processing; S33, Construct an attention modulation module, which adaptively modulates the cross-attention map obtained in S32 based on the layout information obtained in S23. This part is expressed as: (8) (9) In the formula The mask representing the bounding box. This indicates the transpose operation. Indicates the first The proportion of the bounding box region to the entire image; S34, use an image decoder to decode the latent features at iteration step 0, recovering the pixel space from the low-dimensional latent space to obtain the restatement iterations. A high-resolution image, this part is expressed as: (10) S4, construct a semantic text regeneration supervision network, restate the image in S2 after adjustment in S3 using a large image translation model, and perform semantic supervision with the input text to update the latent features of each iteration step in S2; the specific steps of S4 are as follows: S41, Construct a large-scale image translation model, for the first... The generated image of the wheel is restated with text, and this part is expressed as: (11) S42, using the same text encoder as S21, performs the following steps: The content of the round text restatement is encoded, and this part is expressed as: (12) S43, high-level semantic supervision is performed on the restated text vector and the initial text vector to indirectly and implicitly correct incorrect semantically generated content. This part is expressed as: (13) In the formula This indicates the calculation of cosine similarity. Represents logarithmic operations. This indicates a restatement of the loss; S44, utilizing high-level semantic supervision loss, for the first... The diffusion process of the wheel The latent features of the iteration step are updated, and a new round of diffusion is executed; this part is expressed as: (14) S5 uses a pre-trained image decoder to reconstruct an image from the latent features obtained in the last step of S2 after iteration.

2. The text-to-image generation method based on attention modulation and text restatement as described in claim 1, wherein the construction of the large language model prompting framework in step S1 includes the following steps: S11, Design the prompt word framework, which includes instructions that describe the task and specify the response format; S12, input the text content entered by the user into the prompt word frame in S11, and use the large language model to generate object labels and layout information, which is represented as: (15)。

Images