A diffusion model-based layout-to-image generation method that can be generalized

Abstract

The application discloses a generalizable layout-to-image generation method based on a diffusion model, and has the characteristics that the method comprises the following steps: constructing a layout-text-picture dataset conforming to natural language expression; acquiring image features, text features and noise image features; generating semantic layout features for training; calculating loss to update semantic layout encoder parameters; and generating a picture given layout-text information. Compared with the prior art, the application has the ability to generalize other fine-tuned diffusion models, can be directly applied to fine-tuned diffusion models using personalized datasets, can generate personalized pictures through layout, can be used together with other controllable generation methods based on diffusion models, is convenient, has excellent effects, and has a good application prospect.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, in particular to a generalizable layout-to-image generation method based on a diffusion model. BACKGROUND

[0002] Based on the text-to-image diffusion model, people can use any text description to generate rich and high-quality pictures. However, it is difficult to control the image in a fine-grained manner only through the text-to-image diffusion model, that is, the text description can only express the overall semantic of the image, and cannot control the semantic of the partial region of the image through partial text description. This limitation limits the user's control ability over the content of the generated image. For the above problem, an additional control method is needed to enhance the control of the semantic information of the local region, that is, the layout-to-image synthesis task. This task aims to generate multiple objects on different scenes according to a given semantic layout. Early LIS tasks based on semantic masks use adversarial generation networks, such as Pix2Pix, which uses a PatchGAN discriminator and an encoder-decoder generator to synthesize semantic images. Pix2PixHD is an extension of this work, which enhances Pix2Pix through a coarse-to-fine and multi-scale network architecture, achieving high-resolution image synthesis. Affine parameters predicted from the input semantic map are used to adjust the activation in the normalization layer. On the other hand, CC-FPSE and SC-GAN do not modify features, but learn to generate convolution kernels and semantic vectors from semantic maps to adjust generation. OASIS introduces a segmentation-based discriminator to improve the alignment of images with input label maps. These GAN-based methods often face challenges such as unstable convergence and mode collapse.

[0003] With the rise of diffusion models, PITI was the first to use a pre-trained text-to-image generation diffusion model for layout-to-image generation. FreestyleNet achieved open semantic layout-to-image generation through a free-style LIS framework, allowing users to generate images with arbitrary semantics in local regions of the image. Although the above methods have achieved excellent performance, these methods all need to fine-tune or retrain the original model. If this method is adapted to other image generation models in other fields, it needs to be retrained, which has a high computational cost.

[0004] Recently, the feature-guided method has shown excellent performance in the field of layout-to-image generation. Works such as eDiff-l, Dense Diffusion, etc. achieve a coarse-grained picture generation by modifying the attention map. Works such as Masked-Attention, BoxDiff, etc. achieve target detection frame-grained layout-to-image generation by forming a gradient to guide the model sampling through the difference between the attention map and the mask information. This method can be flexibly migrated to different domain models, but this method can only roughly comply with the user input spatial layout.

[0005] In summary, the existing diffusion model for layout-to-image generation needs to fine-tune or retrain the original model. If this method is adapted to other field picture generation models, it needs to be retrained, which has a high computational cost. Although the feature-guided method can be flexibly migrated to different domain models, this method can only roughly comply with the user input spatial layout. SUMMARY

[0006] The purpose of the present application is to provide a generalizable layout-to-image generation method based on a diffusion model to overcome the shortcomings of the prior art. The method uses a layout-text to obtain a semantic layout graph, obtains semantic layout features from the intermediate layer image hidden features of the diffusion model and the semantic layout graph through a semantic layout encoder, and trains the existing conditional diffusion model. Under the given text features and semantic layout features, the noise image features are used to predict the added noise, calculate the loss to update the semantic layout encoder parameters, and generate images according to the given layout-text information. Based on the existing dataset, the method generates a layout-text-image dataset that conforms to natural language expression through a multi-modal large language model. The image features and text features are used for training. The image and text are obtained through an image encoder and a text encoder to obtain image features and text features. The image features are added with noise multiple times to obtain noise image features, and the semantic layout features are generated for training. This method can be directly applied to the diffusion model fine-tuned using personalized datasets. Users can use the present application to generate personalized pictures through layout, and can also use it with other diffusion model-based controllable generation methods. The method is convenient, has excellent effect, and has good application prospect.

[0007] The purpose of the present application is achieved in that a generalizable layout-to-image generation method based on a diffusion model, characterized by using a layout-text to obtain a semantic layout graph method, obtaining semantic layout features of the intermediate layer image hidden features and the semantic layout graph in the diffusion model through a semantic layout encoder, training the diffusion model under the given text features and semantic layout features, predicting the added noise, calculating the loss to update the semantic layout encoder parameters, and realizing the generation of pictures given the layout-text information. The method specifically comprises the following steps:

[0008] Step 1: inputting layout information and picture information into a multimodal pre-training large model to obtain text information , and constructing a layout-text-picture data set conforming to natural language expression further comprising:

[0009] 1-1: obtaining COCO-stuff 2017 data set, for any data in the data set, including layout information and picture information ;

[0010] 1-2: inputting each layout information sub-region , corresponding image information sub-region , and class information of the sub-region to an advanced multimodal large language model to generate a text description closely related to the content of the sub-region and detailed and rich ;

[0011] 1-3: splicing these text descriptions in the order of their layout information to form a text that is coherent, complete and conforms to the habit of natural language expression ;

[0012] 1-4: constructing a training data set containing layout information , picture information and detailed description text .

[0013] Step 2: inputting image information and text information into image encoder VAE and text encoder CLIP-Text Encoder respectively to generate image features and text features , and adding noise to the image features multiple times to obtain noise images satisfying Gaussian distribution for training further comprising:

[0014] 2-1: inputting text The text features are input into the pre-trained CLIP-Text Encoder to obtain text features. ;

[0015] 2-2: Image information The image features are input into a pre-trained image encoder (VAE) to extract image features. ;

[0016] Image features Adding noise multiple times yields pure noise image features. , Defined by the following equation (a):

[0017] (a).

[0018] in, It is a time step Gaussian noise, It is the first The variance of the noise addition is a constant, which increases with... It increases with the increase of; This is the original image. This is the image after noise has been added; It is also a constant, and the calculation formula is: .

[0019] Step 3: Transfer image information Text information Layout information The processed data is then input into the semantic layout encoder to generate semantic layout features. The training further includes:

[0020] 3-1: Selecting features from arbitrary noisy images In the input stable diffusion model, hidden layer features are extracted during the UNet downsampling process. The residual convolutional layer output features in the Transformers module corresponding to sizes 64, 32, 16, and 8. , as latent features of the intermediate layer image;

[0021] 3-2: Text information Encode the data, padded it, and obtain a length of 77. For any Construct a binary mask where 1 indicates that the position should generate a mask similar to the one used in the image. Images with the same semantic meaning, 0 indicates that images should not be generated. Images with the same semantic meaning are ultimately used to obtain a semantic layout of length 77. ; to extract intermediate layer image features with semantic layout input semantic layout features into the semantic layout encoder .

[0022] 3-3: input the intermediate layer image features with semantic layout input semantic layout features into the semantic layout encoder , specifically comprising:

[0023] 3-3-1: for the UNet down-sampled 64, 32, 16, 8 size corresponding The module distribution structure of the semantic layout encoder is the same, specifically including a self-attention layer and a cross-attention layer;

[0024] 3-3-2: input the intermediate layer image features to the self-attention layer of the semantic layout encoder;

[0025] 3-3-3: input the semantic layout features through the attention enhancement module to inject the attention map of the cross-attention layer , specifically comprising:

[0026] 3-3-4: input the image hidden features as the query , input the text hidden features as the key , and obtain the attention map shown in the following formula (b) through the cross-attention mechanism :

[0027] (b).

[0028] 3-3-5: for each token's attention map , obtain its maximum value and minimum value ;

[0029] 3-3-6: for the semantic layout features corresponding to each token, replace the position with a value of 1 with the maximum value obtained in the previous step , and replace the position with a value of 0 with the minimum value obtained in the previous step , to obtain the layout-containing attention map M shown in the following formula (c) layout ;

[0030] M layout (c).

[0031] 3-3-7: inputting the layout-containing attention map M layout replacing the original attention map .

[0032] Step 4: inputting the noise image features , text features , semantic layout features into the diffusion model, calculating the mean square error of the predicted noise and the added noise, and adjusting the model parameters through back propagation, specifically including:

[0033] The mean square error L of the predicted noise and the added noise is calculated by the following formula (d) MSE :

[0034] L MSE (d).

[0035] wherein, is the real added noise, is the final predicted noise.

[0036] Step 5: after the training is completed, test the model, input the given layout-text information into the model, and generate a picture that meets the layout-text information.

[0037] Compared with the prior art, the present application has the ability to promote other fine-tuned diffusion models, can be directly applied to fine-tuned diffusion models using personalized data sets, and users can generate personalized pictures through layout, and the present application can be used together with other controllable generation methods based on diffusion models, the method is convenient, the effect is excellent, and has good application prospect. BRIEF DESCRIPTION OF DRAWINGS

[0038] Figure 1 is a flowchart of the present application;

[0039] Figure 2 is a training flowchart of the layout-to-image generation model:

[0040] Figure 3 is a flowchart of generating pictures from layout and text information. DETAILED DESCRIPTION

[0041] Exemplary embodiments of the present disclosure will be described in greater detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided so that the present disclosure can be more thoroughly understood and the scope of the present disclosure can be accurately conveyed to those skilled in the art.

[0042] First, the noun terms involved in the embodiments are explained.

[0043] Text-to-image diffusion model: Text-to-image diffusion models generate images by gradually denoising them, starting from an initial random noise image and gradually restoring the clear image. During training, the model learns to predict the noise added in each denoising step and gradually remove it. In the generation phase, the generation process is guided by conditional information such as text descriptions, so that the final generated image meets the text requirements. Such models usually use noise prediction networks such as neural networks to handle denoising tasks, and are popular methods in the field of image generation due to their stability and high-quality generation effects. Famous text-to-image diffusion models such as DALL·E 2, Stable Diffusion, and Imagen have achieved remarkable results in multiple fields, including artistic creation, advertising design, game development, etc.

[0044] Layout-to-image generation task: Layout-to-image generation is a key branch of conditional image generation, aiming to accurately generate matching images according to user-specified layout information such as target boxes, semantic segmentation maps, key points, etc. Specifically, given a text information and layout information , there are several entities in the text information , and for any , it corresponds to a layout information sub-region , which is a binary representation. For layout information sub-region , the region with a value of 1 indicates that the image with the specified semantics should be generated in that region. The layout-to-image generation (LIS) task is based on text information and fine-grained layout information to generate an image that meets the input information. Further, the generalizable layout-to-image generation is that by training the layout-to-image generation task on the base diffusion model, the method can be directly transferred to the personalized image generation model after fine-tuning the base diffusion model without fine-tuning.

[0045] Embodiment 1

[0046] Reference Figure 1 ​The application specifically comprises: constructing a layout-text-picture dataset conforming to natural language expression, generating a layout-text-picture dataset conforming to natural language expression on the basis of an existing dataset through a multi-modal large language model; generating image features and text features for training, obtaining image features and text features through an image encoder and a text encoder respectively, and adding noise conforming to a Gaussian distribution to the image features multiple times to obtain noise image features; generating semantic layout features for training, obtaining a semantic layout graph through a layout-text, and obtaining semantic layout features through a semantic layout encoder from image hidden features in an intermediate layer of a diffusion model and the semantic layout graph; training the diffusion model through an existing condition, predicting the added noise under the condition of given text features and semantic layout features, calculating a loss to update parameters of the semantic layout encoder; and generating a picture given the layout-text information. The method specifically comprises the following steps:

[0047] Step S101: generating a layout-text-picture dataset conforming to natural language expression

[0048] The layout-text-picture dataset conforming to natural language expression is constructed on the basis of an existing dataset, and the layout-text-picture dataset conforming to natural language expression is generated through a multi-modal large language model. The specific operation is as follows:

[0049] The COCO-stuff 2017 dataset is obtained, and for any data in the dataset, each layout information sub-region corresponding image information sub-region , and the class information of the sub-region are input into an advanced multi-modal large language model to generate text description closely related to the content of the sub-region and detailed and rich The user can better control the picture generation through text information conforming to the user's expression habit.

[0050] Step S102: obtaining noise image features

[0051] The image is input into an image encoder to generate image features for training , and the image features are added with noise conforming to a Gaussian distribution multiple times to obtain noise image features The specific operation is to input picture information into a pre-trained picture encoder VAE to obtain image features ; the image features are added with noise multiple times to obtain pure noise image features , which is defined by the following formula (a):

[0052] (a).

[0053] wherein, is the time step Gaussian noise, is the th added noise, is a constant, which increases with the increase of ; is the original image, is also a constant, and the calculation formula is: .

[0054] Step S103: obtaining text features by a text text encoder

[0055] The text is input into the pre-trained text encoder CLIP-Text Encoder to obtain text features for training, and the specific operation is to split the text into multiple text descriptions (phrases), and the phrases are respectively input into the text encoder to obtain the encoding features of the text descriptions , and then the encoding features are spliced according to the order of the text descriptions to obtain the final text features text This can well avoid the problem of attribute confusion between different phrases, and ensure that the generated result is faithful to the input text .

[0056] Step S104: obtaining semantic layout features

[0057] The semantic layout features for training are obtained by inputting the diffusion model intermediate layer image hidden features and the semantic layout graph into the semantic layout encoder through the layout-text. Any noise image feature is input into the stable diffusion model, and hidden layer features in the UNet downsampling process are extracted, that is, residual convolution layer output features in the Transformers module corresponding to positions of 64, 32, 16 and 8 sizes , which are used as intermediate layer image hidden features.

[0058] The text information is encoded, and after padding, a with a length of 77 is obtained. For any , a binary mask is constructed, wherein 1 indicates that the position should generate an image with the same semantics as , and 0 indicates that the position should not generate an image with the same semantics as​ Images with the same semantic meaning are ultimately used to obtain a semantic layout of length 77. .

[0059] intermediate layer image features With semantic layout Input into the semantic layout encoder to obtain semantic layout features The advantage of doing this is that it can bind layout information with semantic information together, enabling open semantic layout to image generation.

[0060] Step S105: Train the diffusion model to predict noise in the features of a noisy target image. The training target is the features of the unnoised target image, and the training conditions are image description and semantic layout-guided features. Input the feature information generated in steps S102, S103, and S104 into the diffusion model to predict the added noise.

[0061] Step S106: Calculate the mean square error of the predicted noise and the added noise, and adjust the model parameters. The mean square error L of the predicted noise and the added noise... MSE Calculated by the following formula (d) :

[0062] (d).

[0063] in, The noise was added to the actual sound. This represents the noise in the final prediction.

[0064] The mean square error L MSE The (loss value) updates the model gradient through the backpropagation algorithm.

[0065] This invention does not update the parameters of the diffusion model, but only the parameters of the semantic layout encoder. Compared with previous methods, this ensures that training is performed on the basic semantic space of the diffusion model, making the method generalizable to other fine-tuned diffusion models. This invention is trained on two 4090 graphics cards with a batch size of 40.

[0066] Step S107: Given a text description and image layout, generate the corresponding image.

[0067] See Figure 2 The specific training of the layout to the image generation model includes the following steps:

[0068] Step S201: Transfer text information Encoding using a text encoder

[0069] Text information Manually cut into multiple phrases , and The inputs are respectively fed into the text encoding to obtain the phrase. coding features word For a text encoder, the text can be encoded into a size of (768, 77), where 77 includes a header label, several word labels, a footer label, and padding labels. The encoded features corresponding to the word labels are then extracted. Based on the multiple phrases cut The text is concatenated in sequence, with a header label at the beginning and a tail label and padding labels at the end, to obtain the final text feature. .

[0070] Step S202: Transfer image information Encode using an image encoder, adding noise that conforms to a Gaussian distribution.

[0071] Image information The dimensions are converted to (512, 512, 3), and then input into the image encoder with an encoded size of (64, 64, 4). Image features The obtained image features The noise-adding expression is as follows: Noise is added. Among them, It is a time step Gaussian noise, It is the first The variance of the noise addition is a constant, and it increases with... It increases with the increase of; This is the original image. It is also a constant, and the calculation formula is: .

[0072] This embodiment A total of 1000 noise additions were performed, and 1000 noisy image features were obtained during these 1000 noise addition processes. .

[0073] Step S203: Transfer layout information Encoding via a semantic layout encoder

[0074] Text information Encode the data, padded it, and obtain a length of 77. For any Construct a binary mask where 1 indicates that the position should generate a mask similar to the one used in the image. Images with the same semantic meaning, 0 indicates that images should not be generated. Images with the same semantic meaning are ultimately used to obtain a semantic layout of length 77. .

[0075] intermediate layer image features to a semantic layout input into a semantic layout encoder to obtain semantic layout features .

[0076] Step S204: Predicting added noise

[0077] The diffusion model is trained to predict the noise of the target image features with noise, the training target is the target image features without noise, and the training condition is the picture description and the semantic layout guide features. The feature information generated by steps S102, S103 and S104 is input into the diffusion model to predict the added noise.

[0078] Step S205: Adjusting model parameters

[0079] The mean square error (loss value) of the predicted noise and the added noise is calculated, and the model parameters are adjusted. The mean square error (loss value) L of the predicted noise and the added noise MSE calculated by the following (d) formula:

[0080] (d).

[0081] wherein, is the real added noise, is the final predicted noise;

[0082] The loss value obtained is updated by the back propagation algorithm to update the model gradient. The parameters of the diffusion model are not updated in the present application, but only the parameters of the semantic layout encoder are updated. Compared with the previous method, this can ensure that the present application is trained in the basic semantic space of the diffusion model, so that the method has the ability to be extended to other fine-tuned diffusion models.

[0083] Referring to Figure 3 The layout-to-image generation tool can generate pictures according to the input layout and text information, and the process specifically includes:

[0084] Step S301: User constructs input layout information

[0085] The user needs to pre-set the semantic information to be generated in different regions of the image. Specifically, different pixels can represent different semantic texts, for example, in the form of {(255, 255, 255), text 1}, which means that the position with color (255, 255, 255) should generate text 1, and the semantic layout features are obtained by inputting into the semantic layout encoder .

[0086] Step S302: User constructs input text information

[0087] The user splices the text information of multiple regions preset by the user into a complete text information , input into the text encoder to obtain text features .

[0088] Step S303: generating a picture in line with the user input

[0089] The user obtains the generated picture information , encodes the layout information and the text information input by the user to obtain semantic layout features and text features , and finally generates a picture in line with the user input through a diffusion model of the method.

[0090] The above is only a further description of the present application and is not intended to limit the present application. Any equivalent implementation of the present application shall be included in the scope of the claims of the present application.

Claims

1. A generalizable layout-to-image generation method based on a diffusion model, characterized in that, The method includes the following steps: Step 1: Set the layout information and image information Inputting text information into a multimodal pre-trained large model Construct a layout-text-image dataset that conforms to natural language expression; Step 2: Transfer image information and text information Input the image encoder VAE and the text encoder CLIP-TextEncoder respectively to generate image features. Text features and image features Add noise 1000 times to obtain noisy image features that satisfy a Gaussian distribution for training. ; Step 3: Transfer image information Text information and layout information Input the semantic layout encoder to generate semantic layout features for training. ; Step 4: Feature the noisy image Text features and semantic layout features Input the diffusion model, calculate the predicted noise and the mean square error of the added noise, and adjust the model parameters through backpropagation; Step 5: After training, test the large model by inputting the given layout-text information into the multimodal large language model to generate an image that matches the layout-text information; Step 3 specifically includes: 3.1: Selecting features from arbitrary noisy images The image features of the hidden layer are then input into a stable diffusion model to extract them during the UNet downsampling process. The residual convolutional layer output features in the Transformers module corresponding to sizes 64, 32, 16, and 8. , as latent features of the intermediate layer image; 3.2: Transferring text information Encode the data, padded it, and obtain a length of 77. For any Construct a binary mask where 1 indicates that the position should generate a mask similar to the one used in the image. Images with the same semantic meaning, 0 indicates that the same image should not be generated at that location. Images with the same semantic meaning are ultimately used to obtain a semantic layout of length 77. ; 3.3: Hidden layer image features With semantic layout Input the semantic layout encoder to obtain semantic layout features ; Step 3.3 involves using hidden layer image features. With semantic layout Input the semantic layout encoder to obtain semantic layout features Specifically, it includes: 3.3.1: The sizes corresponding to 64, 32, 16, and 8 for UNet downsampling. The modules are distributed to construct a semantic layout encoder with the same structure, specifically including: a self-attention layer and a cross-attention layer; 3.3.2: Extracting hidden layer image features The self-attention layer of the input semantic layout encoder; 3.3.3: Incorporating semantic layout features An attention map is injected into the cross-attention layer via the attention enhancement module. middle; Step 3.3.3 will use semantic layout features An attention map is injected into the cross-attention layer via the attention enhancement module. Specifically, this includes: 3.3.3.1: Hidden layer image features As a query Using textual latent features as keys Obtain attention map through cross-attention mechanism Its expression is shown in equation (b) below: （b）； 3.3.3.2: Attention Map for Each Token Get its maximum value and minimum value ; 3.3.3.3: Semantic layout features for each token Replace the position with the maximum value obtained in the previous step. Replace the positions with values ​​of 0 with the minimum value obtained in the previous step. The attention map M containing the layout is obtained. layout Its expression is shown in equation (c) below; M layout (c); 3.3.3.4: Attention map containing layout Replace with attention map .

2. The generalizable layout-to-image generation method based on a diffusion model according to claim 1, characterized in that, Step 1 specifically includes: 1.1: Obtaining layout information from the COCO-stuff 2017 dataset and image information Any data, including layout information Divided into two or more contiguous sub-regions Each sub-region Corresponding to a category information ; 1.2: Divide each sub-region Corresponding image information sub-region and the sub-region Category information They are jointly input into a multimodal large language model to generate detailed and rich text descriptions that are closely related to the content of the sub-region. ; 1.3: Text Description By layout information The text is assembled in the order listed to form text information that conforms to the expression habits of natural language. Build a structure containing layout information Image information and text The training dataset.

3. The generalizable layout-to-image generation method based on a diffusion model according to claim 1, characterized in that, Step 2 specifically includes: 2.1: Transfer text information Input is fed into the pre-trained text encoder CLIP-Text Encoder to obtain text features. ; 2.2: Transfer image information Input a pre-trained image encoder (VAE) to obtain image features. ; 2.3: Image Features Adding noise to obtain noisy image features , Defined by the following equation (a): （a）； That For time steps Gaussian noise; Noise weighting coefficient; This is the original image.

4. The generalizable layout-to-image generation method based on a diffusion model according to claim 1, characterized in that, The mean square error L of the predicted noise and the added noise in step 4 MSE Calculated by the following formula (d): L MSE (d); in, Noise added to the actual object; The noise in the final prediction; For time steps.

Images