Diffusion model-based generalizable layout-to-image generation method

Abstract

The invention discloses a method for generalizable layout-to-image generation based on a diffusion model. The method is characterized by comprising the following steps: constructing a layout-text-image data set according with natural language expression; obtaining image features, text features and noise image features; generating semantic layout features for training; calculating loss to update semantic layout encoder parameters; and giving layout-text information to generate a picture and the like. Compared with the prior art, the method has the capability of popularizing other diffusion models subjected to fine tuning, is directly applied to the diffusion models subjected to fine tuning by using a personalized data set, generates personalized pictures through layout, can be used together with other controllable generation methods based on the diffusion models, and is convenient, excellent in effect and high in practicability. Good application prospects are realized.

Description

Technical Field

[0001] The invention relates to the technical field of computer vision, in particular to a generalizable layout-to-image generation method based on a diffusion model. Background Art

[0002] Based on the text-to-image diffusion model, people can use arbitrary text descriptions to generate rich, high-quality images. However, it is difficult to perform fine-grained spatial control of images using only the text-to-image diffusion model, that is, the text description can only express the semantics of the entire image, but cannot control the semantics of a part of the image through a partial text description. This limitation limits the user's ability to control the content of the generated image. For the above problems, additional control methods are needed to enhance the control of semantic information in local areas, namely, the layout-to-image synthesis task. This task aims to generate multiple objects on different scenes based on a given semantic layout. Early semantic mask-based LIS tasks used adversarial generative networks, such as Pix2Pix, which synthesized semantic images through a PatchGAN discriminator and an encoder-decoder generator. Pix2PixHD is a continuation of this work, which enhances Pix2Pix through a coarse-to-fine and multi-scale network architecture to achieve high-resolution image synthesis. The activations in the normalization layer are adjusted using affine parameters predicted from the input semantic map. On the other hand, CC-FPSE and SC-GAN do not modify features, but learn to generate convolution kernels and semantic vectors from semantic maps to regulate generation. OASIS introduces a segmentation-based discriminator to improve the alignment of images with the input label map. These GAN-based methods often face challenges related to unstable convergence and mode collapse.

[0003] With the rise of diffusion models, PITI was the first to use a pre-trained diffusion model for text-to-image generation for layout-to-image generation. FreestyleNet uses the freestyle LIS framework to achieve open semantic layout-to-image generation, allowing users to generate images with arbitrary semantics in local areas of an image. Although the above methods have achieved excellent performance, they all require fine-tuning or retraining of the original model. If this method is adapted to image generation models in other fields, retraining is required, and the computational cost is relatively high.

[0004] Recently, feature-guided methods have shown excellent performance in the field of layout-to-image generation. Works such as eDiff-l and Dense Diffusion have achieved coarse-grained image generation by modifying the attention map. Works such as Masked-Attention and BoxDiff have achieved layout-to-image generation at the granularity of target detection boxes by forming gradient-guided model sampling through the difference between the attention map and the mask information. This method can be flexibly transferred to models in different fields, but this method can only roughly comply with the spatial layout input by the user.

[0005] In summary, the existing diffusion model needs to fine-tune or retrain the original model to generate layouts from images. If this method is adapted to image generation models in other fields, it needs to be retrained, which has a high computational cost. Although the feature-guided method can be flexibly transferred to models in different fields, it can only roughly comply with the spatial layout input by the user. Summary of the invention

[0006] The purpose of the present invention is to provide a generalizable layout-to-image generation method based on a diffusion model to address the deficiencies of the prior art. The method adopts a layout-text method to obtain a semantic layout graph, obtains semantic layout features through a semantic layout encoder from the intermediate layer image hidden features of the diffusion model and the semantic layout graph, trains the diffusion model through existing conditions, predicts the added noise under the given text features and semantic layout features, calculates the loss and updates the semantic layout encoder parameters, and realizes the generation of pictures from given layout-text information. Based on the existing data set, the method generates a layout-text-picture data set that conforms to natural language expression through a multimodal large language model; generates image features and text features for training, obtains image features and text features through an image encoder and a text encoder respectively, and adds noise that conforms to a Gaussian distribution to the image features to obtain noise image features multiple times, and generates semantic layout features for training. The method can be directly applied to a diffusion model that has been fine-tuned using a personalized data set. Users can use the present invention to generate personalized pictures through layout, and can also be used together with other controllable generation methods based on diffusion models. The method is convenient, has excellent effects, and has good application prospects.

[0007] The purpose of the present invention is achieved in this way: a generalizable layout-to-image generation method based on a diffusion model, which is characterized by adopting a layout-text method to obtain a semantic layout map, obtaining semantic layout features through a semantic layout encoder using the latent features of the intermediate layer image of the diffusion model and the semantic layout map, training the diffusion model through existing conditions, predicting the added noise under the given text features and semantic layout features of the noise image features, calculating the loss to update the semantic layout encoder parameters, and realizing the generation of images from given layout-text information. The method specifically includes the following steps:

[0008] Step 1: Layout information and picture information Input multimodal pre-trained large model to obtain text information , building a layout-text-image dataset that conforms to natural language expression further includes:

[0009] 1-1: Get the COCO-stuff 2017 dataset, including layout information for any data in the dataset and picture information ;

[0010] 1-2: Each layout information sub-area Corresponding image information sub-area , and the category information of the sub-area , which are input into an advanced multimodal large language model to generate a detailed and rich text description that is closely related to the content of the sub-region. ;

[0011] 1-3: Describe these texts They are spliced ​​together in the order in which they appear in the layout information to form a coherent, complete text that conforms to natural language expression habits. ;

[0012] 1-4: Build including layout information , picture information and detailed description text The training data set.

[0013] Step 2: Convert image information and text messages Input into the image encoder VAE and text encoder CLIP-Text Encoder respectively to generate image features With text features , and add noise to the image features multiple times to obtain a noise image that satisfies the Gaussian distribution , used for training further includes:

[0014] 2-1: Text Input into the pre-trained text encoder CLIP-Text Encoder to obtain text features ;

[0015] 2-2: Image information Input into the pre-trained image encoder VAE to obtain image features ;

[0016] Image features Add noise multiple times to get pure noise image features , It is defined by the following formula (a):

[0017] (a).

[0018] in, is the time step Gaussian noise, It is The variance of the sub-additive noise is a constant that varies with increases with the increase of is the original image, is the image after adding noise; It is also a constant, and the calculation formula is: .

[0019] Step 3: Convert image information , text information , layout information After processing, it is input into the semantic layout encoder to generate semantic layout features Further training includes:

[0020] 3-1: Select arbitrary noise image features , input into the stable diffusion model, extract the hidden layer features in the UNet downsampling process , the residual convolutional layer output features in the Transformers module corresponding to the sizes 64, 32, 16, and 8 , as the hidden features of the intermediate layer image;

[0021] 3-2: Text information Encode and fill to get a length of 77 , for any Construct a binary mask where 1 indicates that the position should generate the same Images with the same semantics, 0 means that the same Images with the same semantics, finally obtaining a semantic layout with a length of 77 ; The intermediate layer image features With semantic layout Input semantic layout features into the semantic layout encoder .

[0022] 3-3: Intermediate layer image features With semantic layout Input into the semantic layout encoder to obtain semantic layout features , specifically including:

[0023] 3-3-1: The corresponding sizes of 64, 32, 16, and 8 for UNet downsampling The module distribution constructs a semantic layout encoder with the same structure, including self-attention layer and cross-attention layer;

[0024] 3-3-2: Intermediate layer image features Input the self-attention layer of the semantic layout encoder;

[0025] 3-3-3: Semantic layout features Injecting attention maps of cross-attention layers through the attention enhancement module Specifically, it includes:

[0026] 3-3-4: Hidden features of the image As a query , using textual latent features as keys , the attention map shown in the following formula (b) is obtained through the cross attention mechanism :

[0027] (b).

[0028] 3-3-5: Attention map for each token , get its maximum value and minimum value ;

[0029] 3-3-6: Semantic layout features corresponding to each token , replace the position with the value 1 with the maximum value obtained in the previous step , replace the position with the value 0 with the minimum value obtained in the previous step , we get the attention map M containing the layout as shown in the following formula (c): layout ;

[0030] M layout (c).

[0031] 3-3-7: The attention map M containing the layout layout Replaced with the original attention map .

[0032] Step 4: Noise image features , text features , semantic layout features In the input diffusion model, the mean square error of the predicted noise and the added noise is calculated, and the model parameters are adjusted through back propagation, including:

[0033] The mean square error L between the predicted noise and the added noise is calculated by the following formula (d): MSE :

[0034] L MSE (d).

[0035] in, is the actual noise added, is the final predicted noise.

[0036] Step 5: After training, test the model, input the given layout-text information into the model, and generate images that match the layout-text information.

[0037] Compared with the prior art, the present invention has the ability to generalize to other fine-tuned diffusion models and can be directly applied to diffusion models fine-tuned using personalized data sets. Users can generate personalized pictures through layout. At the same time, the present invention can be used together with other controllable generation methods based on diffusion models. The method is convenient, the effect is excellent, and it has good application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 It is a schematic diagram of the process of the present invention;

[0039] Figure 2 Training flow chart for layout to image generation model:

[0040] Figure 3 Generate image flow charts for layout and text information. DETAILED DESCRIPTION

[0041] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0042] First, the terms involved in the embodiments are explained.

[0043] Text-to-image diffusion models: Text-to-image diffusion models generate images by gradually denoising, starting from an initial random noise image and gradually restoring a clear image. During the training process, the model learns how to predict the noise added in each denoising step and gradually removes this noise. In the generation phase, the generation process is guided by conditional information (such as text description) so that the final generated image meets the text requirements. This type of model usually uses noise prediction networks (such as neural networks) to handle denoising tasks, and has become a popular method in the field of image generation due to its stability and high-quality generation effects. Well-known text-to-image diffusion models such as DALL·E 2, Stable Diffusion, and Imagen have achieved remarkable results in many fields, including art creation, advertising design, game development, etc.

[0044] Layout-to-image generation task: Layout-to-image generation is a key branch of conditional image generation, which aims to accurately generate matching images based on user-specified layout information (such as target boxes, semantic segmentation maps, key points, etc.). Specifically, given a text information and layout information , text information There are several entities in , for any Each corresponds to a layout information sub-area , which is a binary representation. For the layout information sub-area , the area with a value of 1 indicates that the image with the specified semantics should be generated in this area. The layout to image generation (LIS) task is based on text information and fine-grained layout information , generating a picture that matches the input information. Furthermore, the generalizable layout-to-image generation is that by training the layout-to-image generation task on the basic diffusion model, the method can be directly transferred to the personalized picture generation model after fine-tuning the basic diffusion model without fine-tuning.

[0045] Example 1

[0046] See also Figure 1, the present invention specifically includes: constructing a layout-text-picture dataset that conforms to natural language expression, and generating a layout-text-picture dataset that conforms to natural language expression through a multimodal large language model based on an existing dataset; generating image features and text features for training, and obtaining image features and text features through an image encoder and a text encoder respectively, and adding noise that conforms to the Gaussian distribution to the image features multiple times to obtain noise image features; generating semantic layout features for training, obtaining a semantic layout graph through layout-text, and obtaining semantic layout features through a semantic layout encoder for the image hidden features of the intermediate layer of the diffusion model and the semantic layout graph; training the diffusion model through existing conditions, and predicting the added noise with the noise image features under the given text features and semantic layout features, and calculating the loss to update the semantic layout encoder parameters; generating pictures given layout-text information. The method specifically includes the following steps:

[0047] Step S101: Generate a layout-text-image dataset that conforms to natural language expression

[0048] Construct a layout-text-image dataset that conforms to natural language expression. Based on the existing dataset, generate a layout-text-image dataset that conforms to natural language expression through a multimodal large language model. The specific operations are as follows:

[0049] Get the COCO-stuff 2017 dataset, and for any data in the dataset, convert each layout information sub-region Corresponding image information sub-area , and the category information of the sub-area , which are input into an advanced multimodal large language model to generate a detailed and rich text description that is closely related to the content of the sub-region. , users can better control image generation through text information that conforms to user expression habits.

[0050] Step S102: Obtain noise image features

[0051] Pass the image through the image encoder to generate image features for training , and the image features Add noise that conforms to Gaussian distribution multiple times to obtain noise image features , its specific operation is to convert the image information Input into the pre-trained image encoder VAE to obtain image features ; Image features Add noise multiple times to get pure noise image features , It is defined by the following formula (a):

[0052] (a).

[0053] in, is the time step Gaussian noise, It is The variance of the secondary noise is a constant that changes with increases with the increase of is the original image, It is also a constant, and the calculation formula is: .

[0054] Step S103: Obtain text features by using the text encoder

[0055] Text Input into the pre-trained text encoder CLIP-Text Encoder to obtain text features for training , its specific operation is to convert the text Split into multiple text descriptions (phrase), the phrase Input them into the text encoding respectively to get the text description The coded features are then described in text The order of splicing is used as the final text feature text This can effectively avoid the problem of attribute confusion between different phrases and ensure that the generated results are faithful to the input text. .

[0056] Step S104: Obtaining semantic layout features

[0057] The semantic layout graph is obtained through layout-text, and the latent features of the intermediate layer image of the diffusion model and the semantic layout graph are passed through the semantic layout encoder to obtain the semantic layout features for training. . Select any noise image feature , input it into the stable diffusion model to extract the hidden layer features in the UNet downsampling process , the residual convolutional layer output features in the Transformers module corresponding to the sizes 64, 32, 16, and 8 , as the hidden features of the intermediate layer image;

[0058] Text message Encode and fill to get a length of 77 , for any Construct a binary mask where 1 indicates that the position should generate the same Images with the same semantics, 0 means that the position should not be generated Images with the same semantics, finally obtaining a semantic layout with a length of 77 .

[0059] The intermediate layer image features With semantic layout Input into the semantic layout encoder to obtain semantic layout features The advantage of doing this is that it can bind layout information with semantic information and realize open semantic layout to image generation.

[0060] Step S105: training a diffusion model to predict the noise of the target image features with noise, the training target is the target image features without noise, and the training conditions are the image description and semantic layout guidance features. The feature information generated by steps S102, S103, and S104 is input into the diffusion model to predict the added noise.

[0061] Step S106: Calculate the mean square error of the predicted noise and the added noise, and adjust the model parameters. MSE Calculated by the following formula (d) :

[0062] (d).

[0063] in, is the actual noise added, is the final predicted noise.

[0064] The mean square error L MSE (Loss value) Update the model gradient through the back-propagation algorithm.

[0065] The present invention does not update the parameters of the diffusion model, but only updates the parameters of the semantic layout encoder. Compared with previous methods, this can ensure that the training is performed on the basic semantic space of the diffusion model, so that the method has the ability to be generalized to other fine-tuned diffusion models. The present invention is trained on two 4090 graphics cards with a training batch of 40.

[0066] Step S107: Generate a corresponding image given a text description and image layout.

[0067] See also Figure 2 ,The specific training of the layout to image generation model includes the following steps:

[0068] Step S201: Transmit text information Encoding via a text encoder

[0069] Text information Manual splitting into phrases , Input them into the text encoding respectively, and get the phrase The encoding features word For the text encoder, the text can be encoded into a size of (768, 77), where 77 includes the head label, several word labels, the tail label and the padding label. Extract the encoding features corresponding to the word label Multiple phrases based on cutting The head tag is filled in the head, the tail tag and the padding tag are filled in the tail, and the final text feature is obtained. .

[0070] Step S202: Image information Encode through the image encoder and add noise that conforms to the Gaussian distribution

[0071] Image information The image is converted to (512, 512, 3) and fed into the image encoder to encode a size of (64, 64, 4). Image features , the image features obtained The noise addition expression is: Add noise. is the time step Gaussian noise, It is The variance of the secondary noise is a constant that changes with increases with the increase of is the original image, It is also a constant, and the calculation formula is: .

[0072] This embodiment of A total of 1000 times, 1000 noise image features are obtained in the 1000 times of noise addition process .

[0073] Step S203: Layout information Encoding via the semantic layout encoder

[0074] Text information Encode and fill to get a length of 77 , for any Construct a binary mask where 1 indicates that the position should generate the same Images with the same semantics, 0 means that the same Images with the same semantics, finally obtaining a semantic layout with a length of 77 .

[0075] The intermediate layer image features With semantic layout Input into the semantic layout encoder to obtain semantic layout features .

[0076] Step S204: predicting the added noise

[0077] The diffusion model is trained to predict the noise of the target image features with noise, the training target is the target image features without noise, and the training conditions are the picture description and the semantic layout guidance features. The feature information generated by steps S102, S103, and S104 is input into the diffusion model to predict the added noise.

[0078] Step S205: Adjust model parameters

[0079] Calculate the mean square error (loss value) between the predicted noise and the added noise, and adjust the model parameters. The mean square error (loss value) L between the predicted noise and the added noise MSE Calculated by the following formula (d):

[0080] (d).

[0081] in, is the actual noise added, is the final predicted noise;

[0082] The obtained loss value is used to update the model gradient through the back propagation algorithm. The present invention does not update the parameters of the diffusion model, but only updates the parameters of the semantic layout encoder. Compared with previous methods, this can ensure that the present invention is trained on the basic semantic space of the diffusion model, so that the method has the ability to be generalized to other fine-tuned diffusion models.

[0083] See also Figure 3 , the generalizable layout to image generation tool generates images based on the input layout and text information, and its specific process includes:

[0084] Step S301: Layout information input by the user

[0085] The user needs to pre-set the semantic information that he wants to generate in different areas of the image. Specifically, different pixels can be used to represent different semantic texts. For example, in the form of {(255, 255, 255), text 1}, it means that the position with color (255, 255, 255) should generate text 1, which is input into the semantic layout encoder to obtain the semantic layout feature. .

[0086] Step S302: Text information input by the user

[0087] The user combines the text information preset in multiple regions into a complete text message , input to the text encoder to obtain text features .

[0088] Step S303: Generate a picture that matches the user input

[0089] The user obtains the generated picture information , based on the layout information and text information encoding input by the user, the semantic layout features are obtained and text features ,Through the diffusion model of this method, a picture that meets the user input is finally generated.

[0090] The above is only a further explanation of the present invention and is not intended to limit the patent of the present invention. Any equivalent implementation of the present invention should be included in the scope of the claims of the patent of the present invention.

Claims

1. A generalizable layout-to-image generation method based on a diffusion model, characterized in that: The method comprises the following steps: Step 1: Layout information and picture information Input multimodal pre-trained large model to obtain text information , build a layout-text-image dataset that conforms to natural language expression; Step 2: Convert image information and text messages Input the image encoder VAE and text encoder CLIP-TextEncoder respectively to generate image features With text features , and the image features Add noise 1000 times to obtain the noise image features that are used for training and satisfy the Gaussian distribution. ; Step 3: Convert image information , text information and layout information Input semantic layout encoder to generate semantic layout features for training ; Step 4: Noise image features , text features and semantic layout features Input the diffusion model, calculate the mean square error of the predicted noise and the added noise, and adjust the model parameters through back propagation; Step 5: After training, test the large model and input the given layout-text information into the multimodal large language model to generate images that match the layout-text information.

2. The generalizable layout-to-image generation method based on a diffusion model according to claim 1, characterized in that: The step 1 specifically includes: 1.1: Get the COCO-stuff 2017 dataset including layout information and picture information Arbitrary data, including layout information Divided into two or more consecutive sub-areas , each sub-region Corresponding to a category information ; 1.2: Each sub-area Corresponding image information sub-area , and the sub-region Category information , and jointly input the multimodal large language model to generate a detailed and rich text description that is closely related to the content of the sub-region ; 1.3: Describe the text By layout information The order in which the text is connected is concatenated to form text information that conforms to the natural language expression habits. , build a file containing layout information , picture information and text The training data set.

3. The generalizable layout-to-image generation method based on a diffusion model according to claim 1, characterized in that: The step 2 specifically includes: 2.1: Text information Input to the pre-trained text encoder CLIP-Text Encoder to obtain text features ; 2.2: Image information Input the pre-trained image encoder VAE to obtain image features ; 2.3: Image features Add noise to get the noise image features , It is defined by the following formula (a): （a）； That is the time step Gaussian noise; is the noise weight coefficient; is the original image.

4. The generalizable layout-to-image generation method based on a diffusion model according to claim 1, characterized in that: The step 3 specifically includes: 3.1: Selecting arbitrary noise image features , input it into the stable diffusion model to extract the hidden layer image features in the UNet downsampling process , the residual convolutional layer output features in the Transformers module corresponding to the sizes 64, 32, 16, and 8 , as the hidden features of the intermediate layer image; 3.2: Text information Encode and fill to get a length of 77 , for any Construct a binary mask where 1 indicates that the position should generate the same Images with the same semantics, 0 means that the position should not be generated Images with the same semantics, finally obtaining a semantic layout with a length of 77 ; 3.3: Hidden layer image features With semantic layout Input semantic layout encoder to obtain semantic layout features .

5. The generalizable layout-to-image generation method based on a diffusion model according to claim 1, characterized in that: The mean square error L of the predicted noise and the noise added in step 4 MSE Calculated by the following formula (d): L MSE (d) in, is the real noise added; is the final predicted noise.

6. The generalizable layout-to-image generation method based on a diffusion model according to claim 4, characterized in that: The step 3.3 is to hide the image features With semantic layout Input semantic layout encoder to obtain semantic layout features , specifically including: 3.3.1: The corresponding sizes of 64, 32, 16, and 8 for UNet downsampling The module distribution constructs a semantic layout encoder with the same structure, including: self-attention layer and cross-attention layer; 3.3.2: Hidden layer image features Input the self-attention layer of the semantic layout encoder; 3.3.3: Semantic layout features Injecting attention maps of cross-attention layers through the attention enhancement module middle.

7. The generalizable layout-to-image generation method based on a diffusion model according to claim 6, characterized in that: The step 3.3.3 converts the semantic layout features Injecting attention maps of cross-attention layers through the attention enhancement module Specifically, it includes: 3.3.3.1: Hidden layer image features As a query , using textual latent features as keys , obtain the attention map through the cross attention mechanism , which is expressed as shown in the following formula (b): （b）； 3.3.3.2: Attention map for each token , get its maximum value and minimum value ; 3.3.3.3: Semantic layout features corresponding to each token , replace the position with the value 1 with the maximum value obtained in the previous step , replace the position with the value 0 with the minimum value obtained in the previous step , get the attention map M containing the layout layout , which is expressed as shown in the following formula (c); M layout (c); 3.3.3.4: Attention map containing layout Replace with attention map .

Images