Image generation method, model training method, device, equipment and storage medium

Abstract

Embodiments of the present application disclose an image generation method, a model training method, a device, a device, and a storage medium. The image generation method includes: when receiving a description text of an image to be generated, randomly generating a noise image; inputting the noise image and the description text into an image generation model to obtain a first predicted noise corresponding to the description text; when the image generation model is trained, network parameters are adjusted based on Gaussian noise added to a sample image, a predicted noise distribution, and a segmentation mask, and the segmentation mask corresponds to entity words and attributes in an image description corresponding to the sample image; generating a target image corresponding to the description text according to the noise image and the first predicted noise. Embodiments of the present application can improve the semantic consistency and picture aesthetics between the generated target image and the description text, and can also improve the image generation efficiency and save storage costs.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of image processing technology, and in particular, to an image generation method, a model training method, a device, a device and a storage medium. Background Art

[0002] With the proposal of the diffusion network, the image generation ability has evolved from initially generating images with specific tags to generating images according to users' text inputs, which has been widely applied in current material generation, creative design and other fields. In these applications, the text-to-image model is used as the basic model base for capacity expansion, including controllable image generation, specific concept enhanced generation, local redrawing, image editing, etc. Therefore, it is extremely important to strengthen the ability of the text-to-image basic model.

[0003] Although the prior art has made great progress in generating high-quality images, there are still great limitations, especially in terms of semantic consistency. The generated images often cannot accurately reflect the content described in the text, that is, the elements in the image do not all correspond to the text description. Currently, common solutions rely on a large image database and complex matching algorithms to ensure semantic consistency between the generated image and the text description. mainly by presetting a large number of image libraries, retrieving the most semantically consistent image through retrieval, or generating multiple images through inference and screening out the best image through voting for return.

[0004] Although the above solutions can improve the image quality to a certain extent, they have great limitations for different user inputs, cannot satisfy both semantic consistency and picture beauty at the same time, and are often limited by the scale and diversity of the database, and the calculation cost is high, especially when dealing with complex scenarios or multi-modal data, which leads to low image generation efficiency. Summary of the Invention

[0005] The embodiments of the present application provide an image generation method, a model training method, a device, a device and a storage medium, which can improve the image generation efficiency on the basis of satisfying semantic consistency and picture beauty.

[0006] To solve the above problems, in a first aspect, the embodiments of the present application provide an image generation method, including:

[0007] When receiving the description text of the image to be generated, randomly generate a noise image;

[0008] Input the noise image and the description text into an image generation model to obtain a first predicted noise corresponding to the description text; when training, the image generation model adjusts network parameters based on Gaussian noise added to a sample image, a predicted noise distribution, and a segmentation mask, and the segmentation mask corresponds to entity words and attributes in the image description corresponding to the sample image;

[0009] Generate a target image corresponding to the description text according to the noise image and the first predicted noise.

[0010] In a second aspect, an embodiment of the present application provides a method for training an image generation model, including:

[0011] Generate an image description of a sample image;

[0012] Determine entity words in the image description and attributes corresponding to the entity words;

[0013] Segment the sample image according to the entity words and the attributes to obtain a segmentation mask corresponding to the entity words and the attributes;

[0014] Overlay Gaussian noise on the sample image to obtain a noise image;

[0015] Input the noise image and the image description into an initial image generation model to obtain a noise distribution corresponding to the noise image and the image description; in the image generation model, fuse a first text feature of the image description and a hidden layer image feature through a cross-attention mechanism;

[0016] Adjust network parameters of the initial image generation model according to the Gaussian noise, the noise distribution, and the segmentation mask to obtain a trained image generation model.

[0017] In a third aspect, an embodiment of the present application provides an image generation device, including:

[0018] A noise image generation module, configured to randomly generate a noise image when receiving a description text of an image to be generated;

[0019] A noise prediction module, configured to input the noise image and the description text into an image generation model to obtain a first predicted noise corresponding to the description text; when training, the image generation model adjusts network parameters based on Gaussian noise added to a sample image, a predicted noise distribution, and a segmentation mask, and the segmentation mask corresponds to entity words and attributes in the image description corresponding to the sample image;

[0020] An image generation module, configured to generate a target image corresponding to the description text according to the noise image and the first predicted noise.

[0021] In a fourth aspect, an embodiment of the present application provides a training device for an image generation model, including:

[0022] An image description generation module, configured to generate an image description of a sample image;

[0023] An entity word attribute determination module, configured to determine entity words in the image description and attributes corresponding to the entity words;

[0024] An image segmentation module, configured to segment the sample image according to the entity words and the attributes to obtain a segmentation mask corresponding to the entity words and the attributes;

[0025] A noise superimposing module, configured to superimpose Gaussian noise on the sample image to obtain a noise image;

[0026] A noise distribution prediction module, configured to input the noise image and the image description into an initial image generation model to obtain a noise distribution corresponding to the noise image and the image description; and fuse a first text feature of the image description with a hidden layer image feature through a cross-attention mechanism in the image generation model;

[0027] A parameter adjustment module, configured to adjust network parameters of the initial image generation model according to the Gaussian noise, the noise distribution, and the segmentation mask to obtain a trained image generation model.

[0028] In a fifth aspect, an embodiment of the present application further provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, the image generation method described in the first aspect is implemented or the training method of the image generation model described in the second aspect is implemented.

[0029] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, the steps of the image generation method described in the first aspect are implemented or the steps of the training method of the image generation model described in the second aspect are implemented.

[0030] The image generation method, model training method, device, equipment, and storage medium provided by the embodiments of the present application generate a noise image when receiving a description text of an image to be generated, input the noise image and the description text into an image generation model to obtain a first predicted noise corresponding to the description text, and generate a target image corresponding to the description text according to the noise image and the first predicted noise. Since the image generation model is trained with segmentation masks corresponding to entity words and attributes, the model can learn image elements corresponding to entity words and attributes, which can improve the semantic consistency and visual aesthetics between the generated target image and the description text. Moreover, there is no need to select the optimal image through a retrieval method or a voting and filtering method, which reduces the computational cost, improves the image generation efficiency, and does not require a preset image library, saving storage costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] To more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings required for use in the embodiments or the description of the prior art. Obviously, the following-described drawings are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0032] Figure 1 is a flowchart of an image generation method provided by an embodiment of the present application;

[0033] Figure 2 is a flowchart of another image generation method provided by an embodiment of the present application;

[0034] Figure 3 is a schematic diagram of the execution process of the image generation method provided by an embodiment of the present application;

[0035] Figure 4 is a flowchart of a training method for an image generation model provided by an embodiment of the present application;

[0036] Figure 5 is a schematic diagram of the process of constructing training data in an embodiment of the present application;

[0037] Figure 6 is a schematic diagram of the joint optimization of some parameters in an embodiment of the present application;

[0038] Figure 7 is a schematic diagram of mask loss optimization in an embodiment of the present application;

[0039] Figure 8 is a schematic diagram of the structure of an image generation device provided by an embodiment of the present application;

[0040] Figure 9It is a schematic structural diagram of a training device for an image generation model provided by an embodiment of the present application;

[0041] Figure 10 It is a schematic structural diagram of an electronic device provided by an embodiment of the present application. Detailed implementation manners

[0042] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0043] Figure 1 It is a flowchart of an image generation method provided by an embodiment of the present application. This image generation method is applicable to scenarios where an image is generated based on a description text. For example, the application scenarios may include creative design scenarios, commercial poster generation scenarios, image editing, and controllable image generation scenarios, etc. As Figure 1 shown, this image generation method includes: Step 110 to Step 130.

[0044] Step 110, when receiving the description text of the image to be generated, randomly generate a noise image.

[0045] Receive the text input by the user for describing the image to be generated, obtain the description text of the image to be generated, and randomly generate a noise image of a preset size. The noise image can be, for example, Gaussian noise.

[0046] Step 120, input the noise image and the description text into an image generation model to obtain a first predicted noise corresponding to the description text; when the image generation model is trained, the network parameters are adjusted based on the Gaussian noise added to the sample image, the predicted noise distribution, and the segmentation mask, and the segmentation mask corresponds to the entity words and attributes in the image description corresponding to the sample image.

[0047] Perform part-of-speech analysis on the description text to determine the entity words and attributes in the description text. Input the part-of-speech analyzed description text and the randomly generated noise image into the image generation model. The image generation model processes the noise image and the description text to predict the noise in the noise image corresponding to the description text, and obtain a first predicted noise corresponding to the description text. The image generation model is a model based on a diffusion network, which is a diffusion model (Diffusion model). The diffusion network can be a residual connection network (Unet network).

[0048] The input layer of the image generation model is used to input a noise image, input the text features corresponding to the description text into the hidden layer of the image generation model, process the noise image through the input layer and the hidden layer, and fuse the output features of the hidden layer with the text features corresponding to the description text through a cross-attention mechanism to guide the generation of the image through the description text.

[0049] When the image generation model is trained, high-quality images are used as sample images, and image descriptions of the sample images are generated through a multi-modal large language model. Entities and attributes in the image descriptions are obtained, and the sample images are segmented based on the entities and attributes to obtain the segmentation masks corresponding to each entity-attribute pair (including an entity and an attribute). The image with Gaussian noise added to the sample image and the image description are input into the initial image generation model. The initial image generation model is used to predict the noise distribution corresponding to the image description in the image with Gaussian noise added, and the network parameters of the image generation model are adjusted based on the Gaussian noise, the noise distribution, and the segmentation masks. During the model training process, through the segmentation masks, the attention scores inside the segmentation masks of the cross-attention mechanism are enhanced, and the attention scores concentrated outside the segmentation masks of the cross-attention are reduced. This can guide the model to learn the image elements corresponding to the entities and attributes during the training process and improve the semantic consistency between the generated image and the description text.

[0050] In an exemplary embodiment, the image generation model includes a diffusion network and a control network. The diffusion network is used to control the picture quality of the image to be generated, and the control network is used to control the semantic consistency between the image to be generated and the description text;

[0051] When training the image generation model, the control network and the diffusion network are alternately trained, and the learning rate of the control network is greater than the learning rate of the diffusion network.

[0052] Among them, the control network, as a low-rank adaptation model, can fine-tune the diffusion network.

[0053] On the basis of training the diffusion network with a small learning rate, a control network based on LoRA (Low-Rank Adaptation, LoRA) is adopted to introduce new parameter variables, and the semantic correlation between the image and the description text is learned with a larger learning rate to enhance the basic semantics of the image generation model, including color attribute changes, size changes, material relationships, spatial relationships, etc., so as to achieve the purpose of improving both the picture beauty and semantic consistency at the same time.

[0054] Step 130, generate the target image corresponding to the description text according to the noise image and the first predicted noise.

[0055] The first predicted noise is a noise distribution map corresponding to the size of the noise image. The difference image between the noise image and the first predicted noise can be calculated, and this difference image is used as the target image corresponding to the description text. Alternatively, the picture quality of this difference image can be further optimized to obtain the target image, so as to further improve the picture quality of the target image.

[0056] Taking an image design website as an example, after the image generation model is trained, the description text input by the user can be passed in through the front end. Through the image generation method provided by the embodiments of the present application, the description text can be visualized, a target image that conforms to the semantics and is exquisite can be generated, and returned to the front end through the interface.

[0057] The image generation method provided by the embodiments of the present application, when receiving the description text of the image to be generated, generates a noise image, inputs the noise image and the description text into the image generation model, obtains the first predicted noise corresponding to the description text, and generates the target image corresponding to the description text according to the noise image and the first predicted noise. Since the image generation model learns the image elements corresponding to the entity words and attributes through the segmentation masks corresponding to the entity words and attributes during training, it can improve the semantic consistency and picture beauty between the generated target image and the description text. Moreover, there is no need to select the optimal image through retrieval or voting filtering methods, reducing the computational cost, improving the image generation efficiency, and there is no need to preset an image library, saving storage costs.

[0058] Figure 2 It is a flowchart of another image generation method provided by the embodiments of the present application. On the basis of the above embodiments, step 130 above is further refined to further optimize the picture quality of the generated target image. As Figure 2 shown, the image generation method includes: step 210 to step 260.

[0059] Step 210, when receiving the description text of the image to be generated, randomly generate a noise image.

[0060] Step 220, input the noise image and the description text into the image generation model, and obtain the first predicted noise corresponding to the description text.

[0061] Among them, the image generation model adjusts the network parameters based on the Gaussian noise added to the sample image, the predicted noise distribution, and the segmentation mask during training, and the segmentation mask corresponds to the entity words and attributes in the image description corresponding to the sample image.

[0062] Step 230, determine the difference between the noise image and the first predicted noise as the initial image corresponding to the description text.

[0063] Subtract the first predicted noise from the noisy image, and use the obtained difference as the initial image corresponding to the description text. Subsequently, the picture quality of this initial image can be further optimized to further improve the picture quality of the finally generated image.

[0064] Step 240: Add the picture quality prompt words that match the description text to the description text to obtain a new description text.

[0065] The above process of generating an initial image based on text description is a text-to-image process. Next, the picture quality of the initial image is optimized through an image generation model, and this process is an image-to-image process.

[0066] The picture quality prompt words can be searched in a pre-set picture quality prompt word library. The picture quality prompt words can be, for example, "high picture quality", "vivid colors", "realistic", "textured", "cinematic texture", etc. Add the picture quality prompt words that match the description text to the description text to obtain a new description text.

[0067] Step 250: Input the initial image and the new description text into the image generation model. The image generation model optimizes the picture quality of the initial image according to the new description text to obtain a second predicted noise corresponding to the initial image and the new description text.

[0068] Input the initial image and the new description text into the image generation model. Input the initial image into the input layer of the image generation model, and input the text features corresponding to the new description text into the hidden layer of the image generation model. Process the initial image through the input layer and the hidden layer. The output features of the hidden layer and the text features corresponding to the new description text are fused through a cross-attention mechanism to guide the image generation model to optimize the picture quality of the initial image through the new description text, and obtain a second predicted noise corresponding to the initial image and the new description text.

[0069] Step 260: Determine the difference between the initial image and the second predicted noise as the target image.

[0070] Subtract the second predicted noise from the initial image, and the obtained difference is the target image corresponding to the description text.

[0071] It should be noted that in the above text-to-image stage, denoising needs to be performed repeatedly multiple times, and the finally obtained image is the initial image. In the above image-to-image stage, denoising also needs to be performed repeatedly multiple times, and the finally obtained image is the target image. The number of denoising times in the image-to-image stage is less than the number of denoising times in the text-to-image stage.

[0072] In the above-mentioned text-to-image stage, Gaussian noise is randomly initialized to obtain a noise image. The description text input by the user is text-featured and input into an image generation model based on a residual connection network together with the noise image. Image generation is performed based on the initialized noise image minus the predicted first predicted noise. This process is repeatedly executed multiple times, for example, 30 times, to generate an initial image with high quality and conforming to the semantic description.

[0073] In the above-mentioned image-to-image stage, the initial image and the text features of the new description text are input into the image generation model, and the image quality of the picture is optimized by subtracting the predicted second predicted noise from the initial image. The entire denoising process is repeatedly executed multiple times, for example, 5 times, to finely tune the picture details for image quality optimization without changing the picture layout.

[0074] The image generation method provided by the embodiments of this application can further improve the aesthetic appearance of the generated target image by generating an initial image based on the description text and the randomly generated noise image, supplementing the description text with a picture quality prompt word to obtain a new description text, and optimizing the image quality of the initial image based on the new description text through the image generation model.

[0075] Based on the above technical solution, before inputting the noise image and the description text into the image generation model, it further includes: determining a target subject word that matches the word in the description text from multiple subject words; superimposing the low-rank adaptation model corresponding to the target subject word onto the image generation model to obtain a new image generation model.

[0076] In the part-of-speech analysis stage, by performing subject word matching on the description text input by the user, if it matches the subject word in the database, this subject word is used as the target subject word. At this time, the low-rank adaptation model corresponding to the target subject word is loaded, and this low-rank adaptation model is superimposed with the base model (i.e., the above-mentioned image generation model) to enhance the base model's understanding of specific input descriptions. The description text obtained after part-of-speech analysis is input into the new image generation model obtained by superimposing the low-rank adaptation model for image inference generation. Among them, the low-rank adaptation model, as a lightweight plug-in, can be superimposed with the base model to fine-tune the base model. LoRA is a lightweight model adjustment technology that realizes model fine-tuning by introducing low-rank decomposition in a pre-trained large model and only adjusting a small number of parameters. LoRA inserts trainable low-rank decomposition matrices in some layers of the pre-trained model, thereby fine-tuning the model to adapt to a new task or dataset without significantly increasing the computational burden.

[0077] By superimposing a low-rank adaptation model that matches the target subject words, the semantic consistency and visual aesthetics of the images finally generated by the image generation model can be further improved.

[0078] Figure 3 It is a schematic diagram of the execution process of the image generation method provided by the embodiments of this application. As Figure 3 shown, the whole process includes: performing part-of-speech analysis for model selection, receiving user input for text-to-image generation to obtain an initial image, screening image quality prompt words, and combining the initial image for image-to-image scene optimization. In the part-of-speech analysis stage, by matching the subject words in the description text input by the user, if it matches the target subject words in the database, a lightweight low-rank adaptation model (Lora model) will be loaded and superimposed with the base model (image generation model) to enhance the base model's understanding of specific input descriptions. The image description obtained after part-of-speech analysis is input into the current base model for text-to-image inference generation. The generation process is basically the same as the training process, except that during training, the sample image with added Gaussian noise as input is changed to random Gaussian noise for random initialization. By text-featurizing the description text input by the user and inputting it together with the Gaussian noise into the image generation model for noise prediction, and subtracting the noise from the Gaussian noise to generate an image. This process is repeated 30 times to generate high-quality images that conform to the semantic description. In the image-to-image stage, random searches for image quality prompt words will be performed in a pre-set image quality prompt word library, such as "high image quality", "vivid colors", "realistic", "textured", and the image quality prompt words will be added to the description text to construct a new description text. The initial image obtained from text-to-image is used to replace the random Gaussian noise as the input of the image generation model for noise prediction. The entire denoising process is repeated 5 times to fine-tune the image details without changing the image layout for image quality optimization.

[0079] Figure 4 It is a flowchart of a training method for an image generation model provided by the embodiments of this application. This training method for the image generation model is used to train the image generation model used in the above embodiments before sampling the image generation method provided in the above embodiments for image generation. As Figure 4 shown, this method may include steps 410 to 460.

[0080] Step 410, generating an image description of a sample image.

[0081] The text of the image description of the sample image can be generated through a large language model.

[0082] In an exemplary embodiment, a large number of high-quality images can be collected as sample images. Through prompt engineering, a multi-modal large language model is used to generate image descriptions. The prompts used can be as follows:

[0083] "1. The description should be no less than 50 words and no more than 500 words. 2. The first sentence of the description should start with a depiction of the main theme of the image. 3. The description should be objective and avoid subjective emotions and speculation. 4. The description should include detailed information about the quantity, color, appearance, and location of the items. 5. The description should follow a logical order, such as from left to right or from the center outwards. 6. Avoid using prefixes such as 'in the picture', 'this image depicts', or 'here is'."

[0084] The process of generating an image description through a multi-modal large language model can be expressed as the following formula:

[0085] Cap = MLLM(I, P pre )

[0086] where P pre represents the pre-set prompts, I represents the sample image to be described, Cap represents the generated image description, and MLLM represents the multi-modal large language model.

[0087] Step 420: Determine the entity words in the image description and the attributes corresponding to the entity words.

[0088] A part-of-speech parsing model can be used for syntactic parsing of the image description. Exemplarily, the Spacy part-of-speech parsing model is used to parse the entity words and adjectives in the sentence. The adjectives are the attributes corresponding to the entity words, and the attributes and entities are bound through a syntactic tree structure to obtain the entity pairs (n) in the description text and the attribute-entity word binding pairs (a-n).

[0089] Step 430: Segment the sample image according to the entity words and the attributes, and obtain a segmentation mask corresponding to the entity words and the attributes.

[0090] A segmentation model can be used to segment the sample image. By inputting the entity words and the sample image into the segmentation model, the segmentation model outputs the corresponding segmentation mask. In the segmentation mask, the segmented main body is white, and the rest of the background is black. Among them, the segmentation model can be, for example, the Segment Anything Model (SAM).

[0091] The process of segmenting the sample image through the segmentation model can be expressed as the following formula:

[0092] M n = SAM(I, n)

[0093] Wherein, I represents a sample image, n represents an entity word, and M n represents a segmentation mask, and SAM represents a segmentation model.

[0094] The segmentation mask corresponding to the entity word can be filtered based on the attributes corresponding to the entity word to determine the segmentation mask jointly corresponding to the entity word and the attributes, that is, to determine the segmentation mask corresponding to the entity word-attribute pair (including the entity word and the attributes).

[0095] In an embodiment of the present application, the method of segmenting the sample image according to the entity word and the attributes to obtain a segmentation mask corresponding to the entity word and the attributes includes: inputting the entity word and the sample image into a segmentation model to obtain one or more initial segmentation masks corresponding to the entity word; for each of the attributes, screening the initial segmentation masks matching the attribute from the one or more initial segmentation masks as the segmentation mask corresponding to the entity word and the attribute.

[0096] Inputting the entity word and the sample image into the segmentation model, and segmenting from the sample image by the segmentation model one or more segmentation masks corresponding to the entity word as the initial segmentation masks of the entity word. When the same entity included in the sample image has multiple different attributes, for the same entity word, the segmentation model can segment a number of segmentation masks equal to the number of attributes, and each segmentation mask corresponds to one attribute. However, the results segmented by the segmentation model cannot distinguish the attributes. At this time, for each attribute, the initial segmentation masks matching the attribute can be screened out from the multiple initial segmentation masks as the segmentation masks corresponding to the entity word and the attributes (entity word-attribute pair).

[0097] By segmenting the sample image based on the segmentation model and screening the segmentation masks matching the attribute based on the attribute, the segmentation masks corresponding to the entity and the attributes can be accurately determined.

[0098] In an embodiment of the present application, the method of screening the initial segmentation masks matching the attribute from the one or more initial segmentation masks for each of the attributes as the segmentation mask corresponding to the entity word and the attribute includes:

[0099] If the entity word corresponds to one of the attributes, determining one initial segmentation mask corresponding to the entity word as the segmentation mask corresponding to the entity word and the attribute;

[0100] If there are multiple attributes corresponding to the entity word, the number of initial segmentation masks is the same as the number of attributes. According to the image features corresponding to each initial segmentation mask and the second text features corresponding to each entity word-attribute pair, determine the segmentation mask corresponding to each entity word-attribute pair. Each entity word-attribute pair includes the entity word and one of the attributes.

[0101] When there is one attribute corresponding to the same entity word determined by the image description, after segmenting the sample image through the segmentation model, a corresponding segmentation mask can be obtained, and this segmentation mask can be directly determined as the segmentation mask corresponding to the entity word and the attribute.

[0102] When there are multiple attributes corresponding to the same entity word determined by the image description, after segmenting the sample image through the segmentation model, segmentation masks with the same number as the number of attributes can be obtained. At this time, feature extraction can be performed on the regions corresponding to each initial segmentation mask in the sample image to obtain the image features corresponding to each initial segmentation mask, and the second text features corresponding to the entity word-attribute pairs are extracted. The image features and the second text features are matched, and the initial segmentation mask with the highest matching degree to an entity word-attribute pair is determined as the segmentation mask corresponding to the entity word-attribute pair.

[0103] In an exemplary embodiment, the determining the segmentation mask corresponding to each entity word-attribute pair according to the image features corresponding to each initial segmentation mask and the second text features corresponding to each entity word-attribute pair includes: for each entity word-attribute pair, determining the similarity between the second text features corresponding to the entity word-attribute pair and the image features corresponding to each initial segmentation mask; determining the initial segmentation mask with the maximum similarity as the segmentation mask corresponding to the entity word-attribute pair.

[0104] Exemplarily, since the segmentation model cannot accurately distinguish attribute concepts, and when there are multiple entities with the same appearance but different attributes in the image, relying solely on the segmentation model cannot accurately distinguish the corresponding entities. The multi-modal perception model CLIP (Contrastive Language-Image Pre-training) can be used as an expert network to judge the relevance between the segmented regions and the corresponding attribute-entity words, and further filtering is performed according to the relevance:

[0105] Emb i =E i (I n ),

[0106] Emb t =E t (a + n),

[0107] sim = cosine(Emb i , Emb t ).

[0108] Among them, I n represents the region corresponding to the entity word n in the sample image I, and E i (I n ) represents obtaining the image features of the region corresponding to the i-th initial segmentation mask corresponding to the entity word n in the sample image I. Emb i represents the image features corresponding to the i-th initial segmentation mask. a represents an attribute, n represents an entity word, and E t (a + n) represents obtaining the second text features corresponding to the entity word n and the attribute a. Emb t represents the second text features corresponding to the entity-attribute pair (a - n). sim represents the similarity between the image features and the second text features.

[0109] If there are multiple identical entities with different attributes in the sample image, it is judged through similarity to determine the segmentation mask corresponding to the entity-attribute pair. This can improve the accuracy of the corresponding relationship between the determined entity-attribute pair and the segmentation mask, thereby improving the semantic consistency between the image generated by the trained model and the description text.

[0110] Through the above method, for a sample image, the corresponding relationship between the image description, segmentation mask, attribute, and entity corresponding to the sample image can be obtained, that is, the sample image-image description-segmentation mask-attribute-entity pair (I, Cap, M n , a, n) is obtained. I represents the sample image, Cap represents the image description, M n represents the segmentation mask, a represents the attribute, and n represents the entity word.

[0111] Figure 5 is the schematic flowchart of constructing training data in the embodiments of this application. As Figure 5 shown, the process of constructing training data includes: for a sample image, first generate the image description of the sample image through a multimodal large language model. Taking the Figure 5 sample image shown as an example, the obtained image description is "a skateboard and a tennis racket and a yellow cup"; perform entity decomposition analysis through a part-of-speech analysis model, and the obtained entity words are "skateboard", "tennis racket", "cup"; perform attribute decomposition analysis through a part-of-speech analysis model, and the obtained entity-attribute pairs are "skateboard", "tennis racket", "yellow cup"; detect the segmentation mask for each entity-attribute pair, and construct a mask label according to the detection result, that is, give the segmentation mask corresponding to each entity-attribute pair.

[0112] Step 440: Add Gaussian noise to the sample image to obtain a noisy image.

[0113] By randomly sampling a timestamp t, gradually add Gaussian noise to the sample image z0(I) to obtain a noisy image, which can be expressed by the following formula:

[0114]

[0115] where ε represents Gaussian noise, and α t represents a coefficient that determines the degree of increased noise, z0(I) represents the sample image, and z t (I) represents the noisy image.

[0116] Step 450: Input the noisy image and the image description into an initial image generation model to obtain a noise distribution corresponding to the noisy image and the image description; in the image generation model, fuse the first text feature of the image description with the hidden layer image feature through a cross-attention mechanism.

[0117] Among them, the image generation model includes a diffusion network and a control network. The diffusion network is used to control the picture quality of the image to be generated, and the control network is used to control the semantic consistency between the image to be generated and the image description; the control network and the diffusion network are alternately trained, and the learning rate of the control network is greater than the learning rate of the diffusion network.

[0118] Use the sample image and image description in the constructed sample image-image description-segmentation mask-attribute-entity pair (I, Cap, M n , a, n) as the input of the image generation model, and preprocess the input. For the text, use the multi-modal perception model CLIP for text feature modeling, which is expressed as follows:

[0119] E t = CLIP(Tokenize(Cap))

[0120] where Tokenize means tokenizing (word segmentation and encoding) the description text Cap to obtain the token (encoding) of the corresponding word, and inputting it into CLIP to obtain the feature vector of the corresponding word token. The feature vectors of all words in the description text are combined into the feature vector E of the description text t .

[0121] To achieve image reconstruction, a diffusion network (Unet network) is used to model the noisy image z t (I) to learn the noise distribution ε superimposed on the sample image z0(I) θ , and compare it with the noisy image z t(I) Subtract to reconstruct the original image (i.e., the sample image). By learning different noise distributions, the diffusion network can reconstruct and generate different high-quality images. To achieve text-guided image generation, the text feature E t is injected into the diffusion network, and information fusion is performed with the intermediate features (hidden layer image features) of the image in the diffusion network through a cross-attention connection-based method. The fusion formula is as follows:

[0122]

[0123] where U i is the hidden layer feature in the diffusion network, W represents the transformation matrix, E t represents the first text feature, U i+1 represents the image feature after text information injection, that is, the fused image feature, and d represents the number of heads of multi-head attention in the cross-attention mechanism.

[0124] During model training, since when the learning rate is large, the generative network (image generation model) is difficult to retain the original aesthetic degree of image generation, and when the learning rate is small, the semantics cannot be effectively improved. Therefore, on the basis of training the diffusion network with a small learning rate, the Lora control network method is adopted to introduce new parameter variables, and the semantic correlation in the text and image is learned at a large learning rate to improve the basic semantics of the model, including color attribute changes, size changes, material relationships, spatial relationships, etc. Specifically, the control network adds a bypass branch network on the basis of the existing diffusion network, and introduces new parameters through operations of dimensionality reduction (MLP matrix A) and dimensionality increase (MLP matrix B) on the input features. In actual training, matrix A is initialized with a random Gaussian distribution, while B is initialized with a 0 matrix to gradually introduce new semantic information.

[0125] The combination method of the diffusion network and the control network can be expressed as follows:

[0126] W + ΔW = W + BA

[0127] h = W + ΔWx = Wx + BAx = (W + BA)x.

[0128] where W represents the diffusion network, B and A are the newly added control networks, x represents the hidden layer input feature, and h represents the hidden layer output feature. During the training of the control network, the parameters of the diffusion network will be fixed to learn the semantic correlation between the constructed text-image pairs.

[0129] To improve both the aesthetic degree of the image and the semantic consistency at the same time, the existing network and the control network are decoupled for alternating training, so that the entire image generation model can focus on one goal for training at the same time, which can greatly reduce the training difficulty and accelerate model convergence.

[0130] Step 460: Adjust the network parameters of the initial image generation model according to the Gaussian noise, the noise distribution, and the segmentation mask to obtain a trained image generation model.

[0131] Based on the Gaussian noise added to the sample image, the predicted noise distribution, and the segmentation mask, adjust the network parameters of the initial image generation model, and iteratively execute the operations of inputting the sample image and the image description, and adjusting the network parameters of the image generation model until the training end condition is met, to obtain a trained image generation model. Among them, the training end condition can be that the number of iterations reaches a preset number, the network parameters converge, or the loss function value converges, etc.

[0132] When alternately training the diffusion network and the control network, after inputting the noise image and the image description corresponding to a sample image into the image generation model, the parameters of the control network can be fixed first, and based on the noise distribution output by the image generation model, as well as the Gaussian noise and the segmentation mask, adjust the network parameters of the diffusion network. Then fix the parameters of this diffusion network, and input the noise image and the image description corresponding to the same sample image into the image generation model again, and based on the noise distribution output by the image generation model, as well as the Gaussian noise and the segmentation mask, adjust the network parameters of the control network. In this way, one round of training is completed, and then other sample images are used for the next round of alternate training.

[0133] In an embodiment of the present application, the adjusting the network parameters of the initial image generation model according to the Gaussian noise, the noise distribution, and the segmentation mask to obtain a trained image generation model includes:

[0134] Determine the mean squared error loss function value of the image generation model according to the Gaussian noise and the noise distribution;

[0135] Determine the mask optimization loss function value according to the segmentation mask. The mask optimization loss function is used to enhance the first attention score inside the segmentation mask of the cross-attention mechanism and reduce the second attention score outside the segmentation mask of the cross-attention mechanism;

[0136] Adjust the network parameters of the initial image generation model according to the mean squared error loss function value and the mask optimization loss function value to obtain a trained image generation model.

[0137] Determine the mean squared error loss function value of the image generation model according to the Gaussian noise and the predicted noise peak according to the following formula:

[0138]

[0139] Among them, L ldm represents the mean squared error loss function value, ∈ represents Gaussian noise, and z t (I) represents the noisy image, t represents the timestamp, and E t represents the first text feature, and ε θ (z t (I), t, E t ) represents the noise distribution predicted by the image generation model.

[0140] By using the mean squared error loss function, the image generation model can be trained to maintain the original picture quality and gradually migrate to a new picture style.

[0141] The joint optimization of some parameters of the image generation model is achieved through the mean squared error loss function. Figure 6 is a schematic diagram of the process of joint optimization of some parameters in the embodiment of the present application. As Figure 6 shown, the autoregressive generative network VAE is used to encode the sample image, Gaussian noise is added to the encoded sample image to obtain a noisy image, and then the noisy image is input into the diffusion network (Unet network) of the image generation model. It is processed through each network layer of the diffusion network, and the image description is input into the hidden layer and the control network of the diffusion network. The diffusion network fuses the first text feature of the image description and the hidden layer image feature through the cross-attention mechanism, and jointly processes the hidden layer image feature in combination with the control network to obtain the predicted noise distribution. Based on this noise distribution and the Gaussian noise added to the sample image, the mean squared error loss function value is determined, and the network parameters of the image generation model are adjusted. The diffusion network and the control network in the image generation model are alternately trained. The specific way of alternate training can refer to the description of the above embodiment and will not be elaborated here.

[0142] Figure 7 is a schematic diagram of mask loss optimization in the embodiment of the present application. As Figure 7 shown, in order to improve semantic consistency, the embodiment of the present application proposes a mask loss optimization process at the entity word level, calculates the loss between the hidden layer feature and the semantic segmentation feature, enhances the understanding and generation of the spatial layout attributes of entity words, and enhances the attention weight in the semantically relevant area and reduces the information interference in the semantically irrelevant area through feature masking and segmentation feature supervision.

[0143] The first text feature is fused with the hidden layer image feature through the cross-attention mechanism at each layer, enabling the image generation model to be guided by text information. During the construction of the cross-attention mechanism, the attention score of the cross-attention mechanism intuitively expresses the importance of the corresponding first text feature in a certain area. The attention score of the cross-attention mechanism is expressed as follows:

[0144]

[0145] Among them, A u represents the attention score and is a two-dimensional matrix of m*n. Here, n represents the number of text tokens (i.e., entity words), and m is the product of the length and width of the image hidden layer features. That is, the value of the i-th row and j-th column of A u represents the probability that the i-th text token appears at the j-th position. Therefore, by constraining a specific token to appear at a specific position and suppressing its appearance in irrelevant regions, that is, enhancing the first attention score inside the segmentation mask of the cross-attention mechanism and reducing the second attention score outside the segmentation mask of the cross-attention mechanism, the semantic consistency can be effectively improved.

[0146] Finally, the overall loss function of the image generation model can be expressed as follows:

[0147] L total = L ldm + L token

[0148] Among them, L total represents the overall loss function value, L ldm represents the mean square error loss function value, and L token represents the mask optimization loss function value.

[0149] In an embodiment of the present application, determining the mask optimization loss function value according to the segmentation mask includes: determining the loss function value inside the mask regarding the first attention score according to the segmentation mask, and determining the loss function value outside the mask regarding the second attention score; determining the sum of the loss function value inside the mask and the loss function value outside the mask as the mask optimization loss function value.

[0150] Among them, the calculation formulas for the loss function value inside the mask and the loss function value outside the mask are as follows:

[0151]

[0152] Among them, represents the loss function value inside the mask, represents the loss function value outside the mask, A u represents the attention score of the cross-attention mechanism, M represents the segmentation mask, A u *M represents the first attention score, and A u *(1 - M) represents the second attention score. A u is the matrix of the overall attention score.

[0153] Using the segmentation mask-attribute-entity pair in the image-image description-segmentation mask-attribute-entity pair (I, Cap, M n , a, n) obtained in the above steps as the fine-grained label. By enhancing the A of the corresponding regions of the attribute and the entity u , and reducing the A corresponding to the irrelevant background u , as an additional optimization objective to assist model training.

[0154] In the calculation formulas of the inner-mask loss function value and the outer-mask loss function value described above, the inner-mask loss function is used to enhance the first cross-attention score inside the mask, where M is the segmentation mask corresponding to the entity and the attribute, 1 is the specified region in the segmentation mask, and 0 is the irrelevant background region. By calculating max(A u *M) to obtain the maximum value of the cross-attention score inside the mask, and pulling it closer to 1, so as to increase the probability of the appearance of the attribute entity inside the mask. The outer-mask loss function is then used to reduce the cross-attention score outside the mask, and suppress the overall distribution by reducing the maximum value of the cross-attention score outside the mask. The final overall mask optimization loss function is as follows:

[0155]

[0156] where, L token represents the mask optimization loss function.

[0157] By introducing this mask optimization loss function, the hidden layer image representation distribution at the token (entity word) level is reallocated to enhance the probability of the appearance of each entity at a specific position. At the same time, the attribute corresponding mask adopted in the embodiments of the present application is the same as the entity mask corresponding to the attribute, and the attribute representation is entityified, greatly enhancing the accuracy of attribute binding and improving semantic consistency.

[0158] The training method of the image generation model provided by the embodiment of the present application determines the entity words in the image description and the attributes corresponding to the entity words by generating the image description of the sample image, segments the sample image according to the entity words and attributes, obtains the segmentation mask corresponding to the entity words and attributes, superimposes Gaussian noise on the sample image to obtain a noise image, inputs the noise image and the image description into the initial image generation model to obtain the noise distribution corresponding to the noise image and the image description, and adjusts the network parameters of the initial image generation model according to the Gaussian noise, the noise distribution and the segmentation mask to obtain the trained image generation model. Since the model learns the image elements corresponding to the entity words and attributes through the segmentation mask corresponding to the entity words and attributes, the semantic consistency and picture aesthetics between the generated target image and the description text can be improved, and there is no need to select the optimal image by means of retrieval or voting filtering, reducing the computational cost, improving the image generation efficiency, and there is no need to preset an image library, saving the storage cost.

[0159] On the basis of the above technical solution, before the step of superimposing Gaussian noise on the sample image to obtain a noise image, it further includes: encoding the sample image and compressing the sample image into a sample image of a preset size.

[0160] Before training the image generation model with the sample image, preprocess the sample image. For the image, in order to reduce the video memory occupancy of the image generation model during training, the variational auto-encoder (VAE) can be used to encode the sample image and compress the sample image to a preset size. The sample image after compression is used as z0(I), and then Gaussian noise can be superimposed. The preset size can be, for example, 128*128*3, and the original size of the sample image can be 1024*1204*3.

[0161] In summary, the embodiments of the present application can be applied to the task of generating images from text for creative image generation and design. The processes of model training and application mainly include: 1. Describing sample images in natural language, detecting and segmenting the mask of the main regions in the sample images, binding and aligning them with natural language, and constructing a fine-grained training dataset. 2. Establishing a training method with joint and alternating optimization of partial parameters to maintain the quality of existing images during model training. 3. Optimizing the mask loss function, supervising the intermediate layer (hidden layer) of text-image interaction, adopting the cross-attention distribution that maximizes the matching of attribute entities within the mask and minimizes the attention distribution outside the mask, encouraging the model to generate specified entities within the mask, and reducing semantic confusion and attention leakage. 4. Using the trained image generation model as the basic model to receive text descriptions as input and quickly generate target images with consistent semantics. The embodiments of the present application effectively improve the semantic consistency of the text-to-image model, reduce the model training time, and lower the cost by constructing a training method for multi-modal block reconstruction.

[0162] The embodiments of the present application effectively solve the problem of the lack of training data for text-to-image generation, provide a general method for generating fine-grained text-image pairs, and generate image-image description-segmentation mask-attribute-entity pairs; propose a semantic consistency alignment training method for block reconstruction, and train the model to achieve more accurate image generation under the condition of limited data, effectively solving the problem that the images generated by the text-to-image model do not match the user input; propose semantic matching generation based on text-to-image and optimize the image quality for image-to-image to solve the problem of insufficient aesthetics of the text-to-image model.

[0163] Figure 8 It is a schematic structural diagram of an image generation device provided by an embodiment of the present application, as Figure 8 shown, the image generation device includes:

[0164] A noise image generation module 810, configured to randomly generate a noise image when receiving a description text of an image to be generated;

[0165] A noise prediction module 820, configured to input the noise image and the description text into an image generation model to obtain a first predicted noise corresponding to the description text; the image generation model adjusts network parameters based on Gaussian noise added to sample images, predicted noise distributions, and segmentation masks during training, and the segmentation masks correspond to entity words and attributes in the image descriptions corresponding to the sample images;

[0166] An image generation module 830, configured to generate a target image corresponding to the description text according to the noise image and the first predicted noise.

[0167] Optionally, the image generation module includes:

[0168] An initial image determination unit for determining the difference between the noise image and the first predicted noise as the initial image corresponding to the description text;

[0169] A text update unit for adding a quality-of-picture prompt word that matches the description text to the description text to obtain a new description text;

[0170] A picture quality optimization unit for inputting the initial image and the new description text into the image generation model, and optimizing the picture quality of the initial image according to the new description text through the image generation model to obtain a second predicted noise corresponding to the initial image and the new description text;

[0171] A target image determination unit for determining the difference between the initial image and the second predicted noise as the target image.

[0172] Optionally, the image generation model includes a diffusion network and a control network. The diffusion network is used to control the picture quality of the image to be generated, and the control network is used to control the semantic consistency between the image to be generated and the description text;

[0173] When training the image generation model, the control network and the diffusion network are alternately trained, and the learning rate of the control network is greater than the learning rate of the diffusion network.

[0174] Optionally, the device further includes:

[0175] A main word matching module for determining a target main word that matches the word in the description text from multiple main words;

[0176] A model update module for superimposing the low-rank adaptation model corresponding to the target main word onto the image generation model to obtain a new image generation model.

[0177] The image generation device provided by the embodiments of the present application is used to implement the steps of the image generation method described in the embodiments of the present application. For the specific implementation manners of the modules of the device, refer to the corresponding steps, which will not be elaborated here.

[0178] The image generation device provided by the embodiment of the present application generates a noise image when receiving a description text of an image to be generated, inputs the noise image and the description text into an image generation model to obtain a first predicted noise corresponding to the description text, and generates a target image corresponding to the description text according to the noise image and the first predicted noise. Since the image generation model is trained with segmentation masks corresponding to entity words and attributes, the model can learn image elements corresponding to entity words and attributes, which can improve the semantic consistency and visual aesthetics between the generated target image and the description text. Moreover, there is no need to select the optimal image through retrieval or voting filtering methods, reducing the computational cost, improving the image generation efficiency, and not requiring a preset image library, which can save storage costs.

[0179] Figure 9 is a schematic structural diagram of a training device for an image generation model provided by an embodiment of the present application, as Figure 9 shown, the training device for the image generation model includes:

[0180] An image description generation module 910, configured to generate an image description of a sample image;

[0181] An entity word attribute determination module 920, configured to determine entity words in the image description and attributes corresponding to the entity words;

[0182] An image segmentation module 930, configured to segment the sample image according to the entity words and the attributes to obtain segmentation masks corresponding to the entity words and the attributes;

[0183] A noise superposition module 940, configured to superimpose Gaussian noise on the sample image to obtain a noise image;

[0184] A noise distribution prediction module 950, configured to input the noise image and the image description into an initial image generation model to obtain a noise distribution corresponding to the noise image and the image description; and fuse first text features of the image description with hidden layer image features through a cross-attention mechanism in the image generation model;

[0185] A parameter adjustment module 960, configured to adjust network parameters of the initial image generation model according to the Gaussian noise, the noise distribution, and the segmentation masks to obtain a trained image generation model.

[0186] Optionally, the parameter adjustment module includes:

[0187] A first loss determination unit, configured to determine a mean square error loss function value of the image generation model according to the Gaussian noise and the noise distribution;

[0188] A second loss determination unit, configured to determine a mask optimization loss function value according to the segmentation mask, where the mask optimization loss function is used to enhance a first attention score inside the segmentation mask of the cross-attention mechanism and reduce a second attention score outside the segmentation mask of the cross-attention mechanism;

[0189] A parameter adjustment unit, configured to adjust network parameters of the initial image generation model according to the mean square error loss function value and the mask optimization loss function value, so as to obtain a trained image generation model.

[0190] Optionally, the second loss determination unit is specifically configured to:

[0191] Determine a loss function value inside the mask for the first attention score and a loss function value outside the mask for the second attention score according to the segmentation mask;

[0192] Determine the sum of the loss function value inside the mask and the loss function value outside the mask as the mask optimization loss function value.

[0193] Optionally, the calculation formulas for the loss function value inside the mask and the loss function value outside the mask are as follows:

[0194]

[0195]

[0196] Wherein, represents the loss function value inside the mask, represents the loss function value outside the mask, A u represents the attention score of the cross-attention mechanism, M represents the segmentation mask, A u *M represents the first attention score, A u *(1 - M) represents the second attention score.

[0197] Optionally, the image segmentation module includes:

[0198] An image segmentation unit, configured to input the entity word and the sample image into a segmentation model to obtain one or more initial segmentation masks corresponding to the entity word;

[0199] A matching unit, configured to, for each of the attributes, screen out an initial segmentation mask that matches the attribute from the one or more initial segmentation masks as the segmentation mask corresponding to the entity word and the attribute.

[0200] Optionally, the matching unit includes:

[0201] The first matching subunit is configured to, if there is one such attribute corresponding to the entity word, determine an initial segmentation mask corresponding to the entity word as the segmentation mask corresponding to the entity word and the attribute.

[0202] The second matching subunit is configured to, if there are multiple such attributes corresponding to the entity word, and the number of initial segmentation masks is the same as the number of attributes, determine the segmentation mask corresponding to each entity word - attribute pair according to the image features corresponding to each initial segmentation mask and the second text features corresponding to each entity word - attribute pair, where each entity word - attribute pair includes the entity word and one such attribute.

[0203] Optionally, the second matching subunit is specifically configured to:

[0204] For each entity word - attribute pair, determine the similarity between the second text features corresponding to the entity word - attribute pair and the image features corresponding to each initial segmentation mask.

[0205] Determine the initial segmentation mask with the maximum similarity as the segmentation mask corresponding to the entity word - attribute pair.

[0206] Optionally, the apparatus further includes:

[0207] An image encoding module, configured to encode the sample image and compress the sample image into a sample image of a preset size.

[0208] Optionally, the image generation model includes a diffusion network and a control network. The diffusion network is used to control the picture quality of the image to be generated, and the control network is used to control the semantic consistency between the image to be generated and the image description.

[0209] The control network and the diffusion network are alternately trained, and the learning rate of the control network is greater than the learning rate of the diffusion network.

[0210] The training apparatus for the image generation model provided by the embodiments of the present application is used to implement the steps of the training method for the image generation model described in the embodiments of the present application. For the specific implementation manners of the modules of the apparatus, refer to the corresponding steps and will not be elaborated here.

[0211] The training device for the image generation model provided by the embodiments of the present application determines the entity words in the image description and the attributes corresponding to the entity words by generating the image description of the sample image, segments the sample image according to the entity words and attributes, obtains the segmentation mask corresponding to the entity words and attributes, superimposes Gaussian noise on the sample image to obtain a noise image, inputs the noise image and the image description into the initial image generation model to obtain the noise distribution corresponding to the noise image and the image description, and adjusts the network parameters of the initial image generation model according to the Gaussian noise, the noise distribution and the segmentation mask to obtain the trained image generation model. Since the model learns the image elements corresponding to the entity words and attributes through the segmentation mask corresponding to the entity words and attributes, the semantic consistency and the picture aesthetics between the generated target image and the description text can be improved, and it is not necessary to select the optimal image by means of retrieval or voting filtering, which reduces the calculation cost, can improve the image generation efficiency, and does not require a preset image library, which can save the storage cost.

[0212] Figure 10 is a schematic structural diagram of an electronic device provided by the embodiments of the present application. As Figure 10 shown, the electronic device 1000 may include one or more processors 1010 and one or more memories 1020 connected to the processors 1010. The electronic device 1000 may further include an input interface 1030 and an output interface 1040 for communicating with another device or system. The program code executed by the processor 1010 may be stored in the memory 1020.

[0213] The processor 1010 in the electronic device 1000 calls the program code stored in the memory 1020 to execute the image generation method or the training method of the image generation model in the above embodiments.

[0214] The embodiments of the present application further provide a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the steps of the image generation method or the training method of the image generation model as described in the embodiments of the present application are implemented.

[0215] The embodiments of the present application further provide a computer program product, and when the computer program product is executed by a processor, the steps of the image generation method or the training method of the image generation model as described in the embodiments of the present application are implemented.

[0216] Each embodiment in this specification is described in a progressive manner. The key point of each embodiment is to describe the differences from other embodiments. The same or similar parts among the embodiments can be referred to each other. For the device embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0217] The above has introduced in detail an image generation method, a model training method, a device, a device and a storage medium provided by the embodiments of the present application. Specific examples are used in this article to elaborate on the principle and implementation manner of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for those of ordinary skill in the art, according to the idea of the present application, there will be changes in the specific implementation manner and application scope. In summary, the content of this specification should not be construed as a limitation to the present application.

[0218] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on such an understanding, the essence of the above technical solution, or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

Claims

1. An image generation method, characterized in that, Including: When receiving the description text of the image to be generated, randomly generate a noise image; Input the noise image and the description text into an image generation model to obtain a first predicted noise corresponding to the description text; when the image generation model is trained, the network parameters are adjusted based on the Gaussian noise added to the sample image, the predicted noise distribution, and the segmentation mask, and the segmentation mask corresponds to the entity words and attributes in the image description corresponding to the sample image; Generate a target image corresponding to the description text according to the noise image and the first predicted noise; Among them, when the image generation model is trained, the network parameters are adjusted based on the Gaussian noise added to the sample image, the predicted noise distribution, and the segmentation mask, including: Determine the mean square error loss function value of the image generation model according to the Gaussian noise and the noise distribution; Determine the mask optimization loss function value according to the segmentation mask, and the mask optimization loss function is used to enhance the first attention score inside the segmentation mask of the cross-attention mechanism and reduce the second attention score outside the segmentation mask of the cross-attention mechanism; Adjust the network parameters of the image generation model according to the Gaussian noise, the noise distribution, and the segmentation mask.

2. The method according to claim 1, wherein The generating a target image corresponding to the description text according to the noise image and the predicted noise includes: Determine the difference between the noise image and the first predicted noise as the initial image corresponding to the description text; Add a picture quality prompt word matching the description text to the description text to obtain a new description text; Input the initial image and the new description text into the image generation model, and the image generation model optimizes the picture quality of the initial image according to the new description text to obtain a second predicted noise corresponding to the initial image and the new description text; Determine the difference between the initial image and the second predicted noise as the target image.

3. The method according to claim 1 or 2, characterized in that, The image generation model includes a diffusion network and a control network. The diffusion network is used to control the picture quality of the image to be generated, and the control network is used to control the semantic consistency between the image to be generated and the description text; When training the image generation model, alternately train the control network and the diffusion network, and the learning rate of the control network is greater than the learning rate of the diffusion network.

4. The method according to claim 1 or 2, characterized in that, Before inputting the noise image and the description text into the image generation model, it further includes: Determine a target subject word that matches the word in the description text from multiple subject words; Overlay the low-rank adaptation model corresponding to the target subject word on the image generation model to obtain a new image generation model.

5. A training method for an image generation model, characterized in that, Including: Generate an image description of the sample image; Determine the entity words in the image description and the attributes corresponding to the entity words; Segment the sample image according to the entity words and the attributes to obtain a segmentation mask corresponding to the entity words and the attributes; Overlay Gaussian noise on the sample image to obtain a noise image; Input the noise image and the image description into an initial image generation model to obtain a noise distribution corresponding to the noise image and the image description; fuse the first text feature of the image description and the hidden layer image feature through a cross-attention mechanism in the image generation model; Adjust the network parameters of the initial image generation model according to the Gaussian noise, the noise distribution, and the segmentation mask to obtain a trained image generation model; The adjusting the network parameters of the initial image generation model according to the Gaussian noise, the noise distribution, and the segmentation mask to obtain a trained image generation model includes: Determine the mean squared error loss function value of the image generation model according to the Gaussian noise and the noise distribution; Determine a mask optimization loss function value according to the segmentation mask, where the mask optimization loss function is used to enhance the first attention score inside the segmentation mask and reduce the second attention score outside the segmentation mask of the cross-attention mechanism; Adjust the network parameters of the initial image generation model according to the mean squared error loss function value and the mask optimization loss function value to obtain a trained image generation model.

6. The method according to claim 5, wherein The determining the mask optimization loss function value according to the segmentation mask includes: Determine a mask internal loss function value regarding the first attention score and a mask external loss function value regarding the second attention score according to the segmentation mask; Determine the sum of the mask internal loss function value and the mask external loss function value as the mask optimization loss function value.

7. The method according to claim 6, wherein The calculation formulas of the mask internal loss function value and the mask external loss function value are as follows: Among them, represents the internal loss function value of the mask, represents the external loss function value of the mask, A u represents the attention score of the cross-attention mechanism, M represents the segmentation mask, A u *M represents the first attention score, A u *(1 - M) represents the second attention score.

8. The method according to any one of claims 5-7, characterized in that The segmenting the sample image according to the entity word and the attribute to obtain a segmentation mask corresponding to the entity word and the attribute includes: Input the entity word and the sample image into a segmentation model to obtain one or more initial segmentation masks corresponding to the entity word; For each attribute, screen out the initial segmentation masks that match the attribute from the one or more initial segmentation masks as the segmentation masks corresponding to the entity word and the attribute.

9. The method according to claim 8, wherein The screening out the initial segmentation masks that match the attribute from the one or more initial segmentation masks as the segmentation masks corresponding to the entity word and the attribute for each attribute includes: If there is one attribute corresponding to the entity word, determine one initial segmentation mask corresponding to the entity word as the segmentation mask corresponding to the entity word and the attribute; If there are multiple attributes corresponding to the entity word, the number of the initial segmentation masks is the same as the number of the attributes. Determine the segmentation masks corresponding to each entity word-attribute pair according to the image features corresponding to each initial segmentation mask and the second text features corresponding to each entity word-attribute pair, where each entity word-attribute pair includes the entity word and one attribute.

10. The method according to claim 9, characterized in that, Determining the segmentation mask corresponding to each of the entity word attribute pairs based on the image features corresponding to each initial segmentation mask and the second text features corresponding to each entity word attribute pair includes: For each of the entity word attribute pairs, determining the similarity between the second text features corresponding to the entity word attribute pair and the image features corresponding to each initial segmentation mask; Determining the initial segmentation mask with the maximum similarity as the segmentation mask corresponding to the entity word attribute pair.

11. The method according to any one of claims 5-7, characterized in that, Before adding the Gaussian noise to the sample image to obtain a noise image, it further includes: Encoding the sample image to compress the sample image into a sample image of a preset size.

12. The method according to any one of claims 5-7, characterized in that The image generation model includes a diffusion network and a control network. The diffusion network is used to control the picture quality of the image to be generated, and the control network is used to control the semantic consistency between the image to be generated and the image description; The control network and the diffusion network are alternately trained, and the learning rate of the control network is greater than the learning rate of the diffusion network.

13. An image generation device, characterized in that, It includes: A noise image generation module, configured to randomly generate a noise image when receiving a description text of an image to be generated; A noise prediction module, configured to input the noise image and the description text into an image generation model to obtain a first predicted noise corresponding to the description text; the image generation model is adjusted based on the Gaussian noise added to the sample image, the predicted noise distribution, and the segmentation mask during training, and the segmentation mask corresponds to the entity words and attributes in the image description corresponding to the sample image; An image generation module, configured to generate a target image corresponding to the description text according to the noise image and the first predicted noise; Wherein, the image generation model is adjusted based on the Gaussian noise added to the sample image, the predicted noise distribution, and the segmentation mask during training, including: Determining the mean square error loss function value of the image generation model according to the Gaussian noise and the noise distribution; Determining a mask optimization loss function value according to the segmentation mask, where the mask optimization loss function is used to enhance the first attention score inside the segmentation mask of the cross-attention mechanism and reduce the second attention score outside the segmentation mask of the cross-attention mechanism; Adjusting the network parameters of the image generation model according to the Gaussian noise, the noise distribution, and the segmentation mask.

14. A training device for an image generation model, characterized in that, It includes: An image description generation module, configured to generate an image description of a sample image; An entity word attribute determination module, configured to determine the entity words in the image description and the attributes corresponding to the entity words; An image segmentation module, configured to segment the sample image according to the entity words and the attributes to obtain a segmentation mask corresponding to the entity words and the attributes; A noise superimposing module, configured to superimpose Gaussian noise on the sample image to obtain a noise image; A noise distribution prediction module, configured to input the noise image and the image description into an initial image generation model to obtain a noise distribution corresponding to the noise image and the image description; and fuse a first text feature of the image description with a hidden layer image feature through a cross-attention mechanism in the image generation model; A parameter adjustment module, configured to adjust network parameters of the initial image generation model according to the Gaussian noise, the noise distribution, and the segmentation mask to obtain a trained image generation model; The parameter adjustment module includes: A first loss determination unit, configured to determine a mean square error loss function value of the image generation model according to the Gaussian noise and the noise distribution; A second loss determination unit, configured to determine a mask optimization loss function value according to the segmentation mask, where the mask optimization loss function is used to enhance a first attention score inside the segmentation mask of the cross-attention mechanism and reduce a second attention score outside the segmentation mask of the cross-attention mechanism; A parameter adjustment unit, configured to adjust network parameters of the initial image generation model according to the mean square error loss function value and the mask optimization loss function value to obtain a trained image generation model.

15. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the image generation method according to any one of claims 1 to 4 or the training method of the image generation model according to any one of claims 5 to 12.

16. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the image generation method according to any one of claims 1 to 4 or the steps of the training method of the image generation model according to any one of claims 5 to 12.

Images