Untrained layout-to-image generation method based on diffusion model

Abstract

The present invention provides a non-trained layout-to-image generation method based on a diffusion model. The present invention designs a non-trained layout-to-image synthesis method for cross-attention guidance, thereby achieving fast and efficient layout generation of images, thereby, for a given layout image and text prompt words, an image can be directly generated.

Description

Technical Field

[0001] The present invention belongs to the technical field of image generation in computer vision, and in particular, relates to a non-training layout-to-image generation method based on a diffusion model. Background Art

[0002] Text-to-image diffusion models, such as DALL-E 2, Imagen, and Stable Diffusion, have demonstrated the ability to synthesize high-quality images given text input. These models are trained on large-scale datasets consisting of image-text pairs. This progress has attracted great attention from the research community and the public, leading to the rise of numerous image editing tools and extending them to various downstream tasks, presenting diversified personalized generation results according to user instructions, simplifying the creative design process, and improving the productivity of professional artists and non-professionals.

[0003] However, despite these successes, there are still challenges in relying solely on textual cues for spatial composition. For example, in the scene design process, the content that users expect to be generated must follow a specified layout, which is not enough to rely on text alone. State-of-the-art image generators also have difficulty correctly conveying accurate spatial composition through text. For location-related descriptive phrases such as "Abicycle on top of a boat", the generator often fails to accurately describe the positional relationship when generating images, and is prone to errors such as blurred and garbled images.

[0004] To address this problem, people have studied layout-to-image synthesis (LIS), which incorporates layout conditions into the generation process and trains a completely new layout-to-image model. However, this requires a large amount of paired layout image training data, which is expensive and challenging to obtain, and does not guarantee the generalization of the new model architecture. In addition, training and fine-tuning the model require a lot of computation.

[0005] A recent study shows that the self-attention map and cross-attention map of the diffusion model have rich structural information, and the cross-attention map can be manipulated by inputting the layout to achieve training-free implementation of layout to image. However, although the training-free implementation of layout to image is achieved, they still have three key problems: 1) "semantic inconsistency", the object required by the user is not generated; 2) "object fusion", the required objects are incorrectly merged; 3) "spatial error", the object required by the user deviates from the layout space, and the synthetic image and layout information are not accurately controlled.

[0006] It can be seen that how to accurately generate an image of a text target at a specified spatial position given a layout image and a text prompt word without prior data is a technical problem that needs to be further solved. Summary of the invention

[0007] To solve the above technical problems, the present invention provides a method for generating an image from a layout without training based on a diffusion model, the method comprising the following steps: Step 1: Obtain a layout image and text prompt words for image generation, and use a Clip model encoder to encode the text prompt words into a text vector.

[0008] Step 2: Input the text vector, layout image and randomly generated Gaussian noise into the pre-trained stable diffusion model, extract the area where the annotation box in the layout image is located, use a soft mask in the cross-attention map of the stable diffusion model to obtain the base class mask map, and obtain all the calculation results of the cross-attention function and the cross-attention map.

[0009] Step 3: Extract the mask corresponding to the cross-attention function and the core word position, apply edge extraction, and use the Sobel operator to maximize the variance of the cross-attention map in the edge area to achieve the maximum edge area.

[0010] Step 4: Extract the cross-attention map, perform weighted average calculation on the cross-attention maps generated by the text prompt words and the weighting factor to obtain the target actual cross-attention map (enriched semantic information).

[0011] Step 5: Calculate the mean square error between the target actual cross-attention map and the actual attention map, modify the noise sample in each denoising step to minimize the loss using gradient descent, and select the result that best fits the layout image and text semantics as the final generated result.

[0012] Optionally, before step 1, the method further includes: downloading an open source text-to-image generator based on a stable diffusion model and a pre-trained Clip model encoder.

[0013] Optionally, in step 1, the given layout image Depend on The bounding box is composed of a given text prompt word Include text phrases corresponding to the bounding box; The position and size of the annotation box are determined by Affected by Used to describe the bounding box The required object in the text prompt word is encoded using the pre-trained Clip model encoder to obtain the text vector ;in, is the embedding dimension.

[0014] Optionally, step 2 includes: inputting the text vector, layout image and randomly generated Gaussian noise into a pre-trained stable diffusion model, and after denoising by the Unet network, Step Cross Attention Map The formula is: ; Among them, query From the given size The key is calculated in the feature map of By text vector It is obtained through linear mapping; Attention map , extract the corresponding text token according to the text token of interest and ; Represents the strength of association of a text token with each spatial position in the feature map.

[0015] Furthermore, in step 2, the randomly generated Gaussian noise is of size Pure noise image, and Respectively represent The height and width of the image; each bounding box Transformed into its corresponding binary mask , a soft mask is used in the cross-attention map of the stable diffusion model ; Get the base class mask map , thus obtaining the loss function ;in, , .

[0016] Optionally, in step 3, the variance of the cross attention map in the edge region is maximized, and the calculation method is: ,in, Represents maximizing the variance of the cross-attention map within the edge region; represents the Sobel operator, represents the variance function.

[0017] Optionally, in step 4, the cross-attention graph weighted calculation method is: ;in, Represents the cross-attention map generated by the text prompt word corresponding to the text start mark [SOT] token, Represents the cross-attention graph generated by the text prompt word corresponding to the end-of-text marker [EOT] token, Expressed as a weighting factor.

[0018] Optionally, in step 5, the generated target cross-attention map is calculated With the actual cross attention map The mean square error between: ;in, Represents the target cross attention map With the actual cross attention map The mean square error between is the total number of cross-attention maps; in the denoising step using stable diffusion, the image features at the current moment are obtained by subtracting the noise vector at the current moment from the image features at the previous moment, which can be expressed as: ;in, Represents the image features at the current moment, Represents the image features of the previous moment; is the factor that controls the strength of the guidance, represents the noise schedule; As the overall constraint, , is a hyperparameter; Used to balance the correlation between cross-attention maps of different objects, is the factor that controls the strength of the guidance.

[0019] The beneficial effects of the present invention are as follows: the present invention utilizes the text-to-image generation model to obtain layout-to-image generation, designs a spatial perception attention module, and uses a soft mask in the cross-attention to generate a base class mask map to achieve spatial constraints on the target object and accurately place it in the specified area. At the same time, edge extraction is applied to maximize the variance of the cross-attention map in the edge area to balance the correlation of the cross-attention maps of different objects. Finally, a token weight module is designed to refine the image content and fill in the semantic information ignored in the denoising process to ensure the consistency of the text features with the generated image. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0021] Figure 1 It is a flowchart of a method for generating an image from a layout without training based on a diffusion model disclosed in an embodiment of the present invention. DETAILED DESCRIPTION

[0022] The following is a description of the implementation of the present application by specific specific embodiments. People familiar with the technology can easily understand other advantages and effects of the present application from the contents disclosed in this specification. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0023] In addition, the technical features involved in the different embodiments of the present application described below can be combined with each other as long as they do not conflict with each other.

[0024] like Figure 1 As shown, an embodiment of the present invention discloses a method for generating an image from a layout without training based on a diffusion model, the method comprising the following steps: Step 1: Obtain a layout image and text prompt words for image generation, and use a Clip model encoder to encode the text prompt words into a text vector.

[0025] Step 2: Input the text vector, layout image and randomly generated Gaussian noise into the pre-trained stable diffusion model, extract the area where the annotation box in the layout image is located, use a soft mask in the cross-attention map of the stable diffusion model to obtain the base class mask map, and obtain all the calculation results of the cross-attention function and the cross-attention map.

[0026] Step 3: Extract the mask corresponding to the cross-attention function and the core word position, apply edge extraction, and use the Sobel operator to maximize the variance of the cross-attention map in the edge area to achieve the maximum edge area.

[0027] Step 4: Extract the cross-attention map, perform weighted average calculation on the cross-attention maps generated by the text prompt words and the weighting factor to obtain the target actual cross-attention map (enriched semantic information).

[0028] Step 5: Calculate the mean square error between the target cross-attention map and the actual cross-attention map, modify the noise sample in each denoising step to minimize the loss using gradient descent, and select the result that best fits the layout image and text semantics as the final generated result.

[0029] In this implementation, considering the role of the text-to-image generation model based on the stable diffusion model in understanding the relationship between text and image and the role of layout constraints in spatial positioning, a training-free layout-to-image synthesis method for cross-attention guidance is designed to achieve fast and efficient layout generation of images. Therefore, for a given layout image and text prompt words, an image can be directly generated.

[0030] Optionally, before step 1, the method further includes: downloading an open source text-to-image generator based on a stable diffusion model and a pre-trained Clip model encoder.

[0031] In this implementation, the Clip model encoder is a frozen encoder.

[0032] Optionally, in step 1, the given layout image Depend on The bounding box is composed of a given text prompt word Include text phrases corresponding to the bounding box; The position and size of the annotation box are determined by Affected by Used to describe the bounding box The required object in the text prompt word is encoded using the pre-trained Clip model encoder to obtain the text vector ;in, is the embedding dimension.

[0033] Optionally, step 2 includes: inputting the text vector, layout image and randomly generated Gaussian noise into a pre-trained stable diffusion model, and after denoising by the Unet network, Step Cross Attention Map The formula is: ; Among them, query From the given size The key is calculated in the feature map of By text vector It is obtained through linear mapping; Attention map , extract the corresponding text token according to the text token of interest and ; Represents the strength of association of a text token with each spatial position in the feature map.

[0034] Furthermore, in step 2, the randomly generated Gaussian noise is of size Pure noise image, and Respectively represent The height and width of the image; each bounding box Transformed into its corresponding binary mask , a soft mask is used in the cross-attention map of the stable diffusion model ; Get the base class mask map , thus obtaining the loss function ;in, , .

[0035] Optionally, in step 3, the variance of the cross attention map in the edge region is maximized, and the calculation method is: ,in, Represents maximizing the variance of the cross-attention map within the edge region; represents the Sobel operator, represents the variance function.

[0036] Optionally, in step 4, the cross-attention graph weighted calculation method is: ;in, Represents the cross-attention map generated by the text prompt word corresponding to the text start mark [SOT] token, Represents the cross-attention graph generated by the text prompt word corresponding to the end-of-text marker [EOT] token, Expressed as a weighting factor.

[0037] Optionally, in step 5, the generated target cross-attention map is calculated With the actual cross attention map The mean square error between: ;in, Represents the target cross attention map With the actual cross attention map The mean square error between is the total number of cross-attention maps; in the denoising step using stable diffusion, the image features at the current moment are obtained by subtracting the noise vector at the current moment from the image features at the previous moment, which can be expressed as: ;in, Represents the image features at the current moment, Represents the image features of the previous moment; is the factor that controls the strength of the guidance, represents the noise schedule; As the overall constraint, , is a hyperparameter; Used to balance the correlation between cross-attention maps of different objects, is the factor that controls the strength of the guidance.

[0038] Compared with the existing methods, the method of the present invention reduces the occupation of memory resources and improves the overall image quality while achieving layout to image generation.

[0039] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0040] Although the embodiments or examples of the present disclosure have been described with reference to the accompanying drawings, it should be understood that the above-mentioned methods, systems and devices are merely exemplary embodiments or examples, and the scope of the present disclosure is not limited by these embodiments or examples, but is only limited by the claims after authorization and their equivalent scope. Various elements in the embodiments or examples may be omitted or replaced by their equivalent elements. In addition, each step may be performed in an order different from that described in the present disclosure. Further, the various elements in the embodiments or examples may be combined in various ways. It is important that with the evolution of technology, many of the elements described herein may be replaced by equivalent elements that appear after the present disclosure.

Claims

1. A non-trained layout-to-image generation method based on a diffusion model, characterized in that: The method comprises the following steps: Step 1: Get the layout image and text prompt words to be generated, and use the Clip model encoder to encode the text prompt words into text vectors; Step 2: Input the text vector, layout image and randomly generated Gaussian noise into the pre-trained stable diffusion model, extract the area where the annotation box is located in the layout image, use a soft mask in the cross-attention map of the stable diffusion model to obtain the base class mask map, and obtain all the calculation results of the cross-attention function and the cross-attention map; Step 3: Extract the mask corresponding to the cross-attention function and the core word position, apply edge extraction, and use the Sobel operator to maximize the variance of the cross-attention map in the edge area to achieve the maximum edge area; Step 4: Extract the cross-attention map, perform weighted average calculation on the cross-attention maps generated by the text prompt words and the weighting factor to obtain the target actual cross-attention map; Step 5: Calculate the mean square error between the target actual cross-attention map and the actual attention map, modify the noise sample in each denoising step to minimize the loss using gradient descent, and select the result that best fits the layout image and text semantics as the final generated result; Step 2 includes: inputting the text vector, layout image and randomly generated Gaussian noise into the pre-trained stable diffusion model, and after denoising by the Unet network, Step Cross Attention Map The formula is: ; Among them, query From the given size The key is calculated in the feature map of By text vector It is obtained through linear mapping; Attention map , extract the corresponding text token according to the text token of interest and ; represents the strength of association of a text token with each spatial position in the feature map; In step 2, the randomly generated Gaussian noise is of size Pure noise image, and Respectively represent The height and width of the image; each bounding box Transformed into its corresponding binary mask , a soft mask is used in the cross-attention map of the stable diffusion model , get the base class mask map , thus obtaining the loss function ,in, , .

2. The method for generating an image from a layout without training based on a diffusion model according to claim 1, characterized in that: Before step 1, the method further includes: downloading an open source text-to-image generator based on a stable diffusion model and a pre-trained Clip model encoder.

3. The method for generating an image from a layout without training based on a diffusion model according to claim 2, characterized in that: In step 1, the given layout image Depend on The bounding box is composed of a given text prompt word Include text phrases corresponding to the bounding box; The position and size of the annotation box are determined by Affected by Used to describe the bounding box The required object in the text prompt word is encoded using the pre-trained Clip model encoder to obtain the text vector ;in, is the embedding dimension.

4. The method for generating an image from a layout without training based on a diffusion model according to claim 1, characterized in that: In step 3, the variance of the cross attention map in the edge region is maximized, and the calculation method is: ;in, Represents maximizing the variance of the cross-attention map within the edge region; represents the Sobel operator, represents the variance function.

5. The method for generating an image from a layout without training based on a diffusion model according to claim 4, characterized in that: In step 4, the cross-attention graph weighted calculation method is: ;in, Represents the cross-attention map generated by the text prompt word corresponding to the text start mark [SOT] token, Represents the cross-attention graph generated by the text prompt word corresponding to the end-of-text marker [EOT] token, Expressed as a weighting factor.

6. The method for generating an image from a layout without training based on a diffusion model according to claim 5, characterized in that: In step 5, the generated target cross attention map is calculated With the actual cross attention map The mean square error between: ;in, Represents the target cross attention map With the actual cross attention map The mean square error between is the total number of cross-attention maps; in the denoising step using stable diffusion, the image features at the current moment are obtained by subtracting the noise vector at the current moment from the image features at the previous moment, which can be expressed as: ;in, Represents the image features at the current moment, Represents the image features of the previous moment; is the factor that controls the strength of the guidance, represents the noise schedule; As the overall constraint, , is a hyperparameter; Used to balance the correlation of cross-attention maps between different objects, is the factor that controls the strength of the guidance.

Images