Image generation method and device, model training method and device, equipment and storage medium

Abstract

The embodiment of the invention discloses an image generation method and device, a model training method and device, equipment and a storage medium, and the image generation method comprises the steps: randomly generating a noise image when a description text of a to-be-generated image is received; inputting the noise image and the description text into an image generation model to obtain first prediction noise corresponding to the description text; network parameters of the image generation model are adjusted based on Gaussian noise added into a sample image, predicted noise distribution and segmentation masks during training, and the segmentation masks correspond to entity words and attributes in image description corresponding to the sample image; and generating a target image corresponding to the description text according to the noise image and the first prediction noise. According to the embodiment of the invention, the semantic consistency and picture aesthetics between the generated target image and the description text can be improved, the image generation efficiency can be improved, and the storage cost can be saved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of image processing technology, and in particular to an image generation method, a model training method, an apparatus, a device and a storage medium. Background Art

[0002] With the introduction of diffusion networks, image generation capabilities have evolved from generating images with specific labels to generating images based on user text input, which is widely used in current material generation, creative design and other fields. In these applications, the text graph model is used as a basic model base to expand capabilities, including controllable image generation, specific concept reinforcement generation, local redrawing, image editing, etc. Therefore, it is extremely important to enhance the capabilities of the text graph basic model.

[0003] Although the existing technology has made great progress in generating high-quality images, it still has great limitations, especially in terms of semantic consistency. The generated images often cannot accurately reflect the content of the text description, that is, not all elements in the image correspond to the text description. At present, common solutions rely on huge image databases and complex matching algorithms to ensure the semantic consistency between the generated image and the text description, mainly by presetting a large number of image libraries, matching the most semantically consistent images through retrieval, or generating multiple images through reasoning, and filtering the best images through voting to return.

[0004] Although the above scheme can improve the image quality to a certain extent, it has great limitations for different user inputs and cannot simultaneously meet the requirements of semantic consistency and picture aesthetics. It is also often limited by the scale and diversity of the database and has a high computational cost, which is especially evident when processing complex scenes or multimodal data, resulting in low image generation efficiency. Summary of the invention

[0005] The embodiments of the present application provide an image generation method, a model training method, an apparatus, a device, and a storage medium, which can improve the efficiency of image generation while satisfying semantic consistency and aesthetics.

[0006] In order to solve the above problems, in a first aspect, an embodiment of the present application provides an image generation method, comprising:

[0007] When receiving a description text of an image to be generated, randomly generating a noise image;

[0008] Inputting the noise image and the description text into an image generation model to obtain a first predicted noise corresponding to the description text; during training, the image generation model adjusts network parameters based on the Gaussian noise added to the sample image, the predicted noise distribution, and the segmentation mask, wherein the segmentation mask corresponds to entity words and attributes in the image description corresponding to the sample image;

[0009] A target image corresponding to the description text is generated according to the noise image and the first predicted noise.

[0010] In a second aspect, an embodiment of the present application provides a method for training an image generation model, comprising:

[0011] Generate image descriptions of sample images;

[0012] Determining entity words in the image description and attributes corresponding to the entity words;

[0013] Segmenting the sample image according to the entity word and the attribute to obtain a segmentation mask corresponding to the entity word and the attribute;

[0014] Superimposing Gaussian noise onto the sample image to obtain a noise image;

[0015] Inputting the noise image and the image description into an initial image generation model to obtain a noise distribution corresponding to the noise image and the image description; fusing the first text feature of the image description with the hidden layer image feature in the image generation model through a cross-attention mechanism;

[0016] According to the Gaussian noise, the noise distribution and the segmentation mask, the network parameters of the initial image generation model are adjusted to obtain a trained image generation model.

[0017] In a third aspect, an embodiment of the present application provides an image generating device, including:

[0018] A noise image generation module, used for randomly generating a noise image when receiving a description text of an image to be generated;

[0019] A noise prediction module, used for inputting the noise image and the description text into an image generation model to obtain a first predicted noise corresponding to the description text; the image generation model is trained by adjusting network parameters based on Gaussian noise added to the sample image, the predicted noise distribution and the segmentation mask, and the segmentation mask corresponds to the entity words and attributes in the image description corresponding to the sample image;

[0020] An image generation module is used to generate a target image corresponding to the description text according to the noise image and the first predicted noise.

[0021] In a fourth aspect, an embodiment of the present application provides a training device for an image generation model, comprising:

[0022] An image description generation module, used to generate image descriptions of sample images;

[0023] An entity word attribute determination module, used to determine the entity words in the image description and the attributes corresponding to the entity words;

[0024] An image segmentation module, used to segment the sample image according to the entity word and the attribute, and obtain a segmentation mask corresponding to the entity word and the attribute;

[0025] A noise superposition module, used for superimposing Gaussian noise onto the sample image to obtain a noise image;

[0026] A noise distribution prediction module is used to input the noise image and the image description into an initial image generation model to obtain a noise distribution corresponding to the noise image and the image description; and to fuse the first text feature of the image description with the hidden layer image feature in the image generation model through a cross-attention mechanism;

[0027] A parameter adjustment module is used to adjust the network parameters of the initial image generation model according to the Gaussian noise, the noise distribution and the segmentation mask to obtain a trained image generation model.

[0028] In the fifth aspect, an embodiment of the present application also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the image generation method as described in the first aspect is implemented or the image generation model training method as described in the second aspect is implemented.

[0029] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the image generation method as described in the first aspect or the steps of the image generation model training method as described in the second aspect.

[0030] The image generation method, model training method, device, equipment and storage medium provided in the embodiments of the present application generate a noise image upon receiving a description text of an image to be generated, input the noise image and the description text into an image generation model, obtain a first predicted noise corresponding to the description text, and generate a target image corresponding to the description text based on the noise image and the first predicted noise. Since the image generation model learns the image elements corresponding to the entity words and attributes through the segmentation masks corresponding to the entity words and attributes during training, the semantic consistency and aesthetics of the generated target image and the description text can be improved, and there is no need to select the optimal image through retrieval or voting filtering, which reduces the computing cost and can improve the image generation efficiency. There is no need to preset an image library, which can save storage costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0032] Figure 1 is a flow chart of an image generation method provided in an embodiment of the present application;

[0033] Figure 2 is a flow chart of another image generation method provided in an embodiment of the present application;

[0034] Figure 3 is a schematic diagram of the execution process of the image generation method provided in an embodiment of the present application;

[0035] Figure 4 It is a flowchart of a training method of an image generation model provided in an embodiment of the present application;

[0036] Figure 5 is a schematic diagram of the process of constructing training data in an embodiment of the present application;

[0037] Figure 6 It is a schematic diagram of the process of joint optimization of some parameters in the embodiment of the present application;

[0038] Figure 7 is a schematic diagram of mask loss optimization in an embodiment of the present application;

[0039] Figure 8 is a structural schematic diagram of an image generating device provided in an embodiment of the present application;

[0040] Fig. 9It is a structural schematic diagram of a training device for an image generation model provided in an embodiment of the present application;

[0041] Fig.10 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0042] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0043] Figure 1 1 is a flowchart of an image generation method provided by an embodiment of the present application. The image generation method is applicable to scenarios where corresponding images are generated based on description texts. For example, application scenarios may include creative design scenarios, commercial poster generation scenarios, image editing, and controllable image generation scenarios. Figure 1 As shown, the image generating method includes: step 110 to step 130.

[0044] Step 110: upon receiving a description text of an image to be generated, randomly generate a noise image.

[0045] A text describing the image to be generated input by a user is received, the description text of the image to be generated is obtained, and a noise image of a preset size is randomly generated. The noise image may be, for example, Gaussian noise.

[0046] Step 120, input the noise image and the description text into an image generation model to obtain a first predicted noise corresponding to the description text; during training, the image generation model adjusts network parameters based on the Gaussian noise added to the sample image, the predicted noise distribution, and the segmentation mask, and the segmentation mask corresponds to the entity words and attributes in the image description corresponding to the sample image.

[0047] The description text is subjected to part-of-speech analysis to determine entity words and attributes in the description text, and the description text after part-of-speech analysis and the randomly generated noise image are input into the image generation model, and the noise image and the description text are processed by the image generation model to predict the noise corresponding to the description text in the noise image, and obtain the first predicted noise corresponding to the description text. The image generation model is a model based on a diffusion network, which is a diffusion model (Diffusion model), and the diffusion network can be a residual connection network (Unet network).

[0048] The input layer of the image generation model is used to input the noisy image, and the text features corresponding to the description text are input into the hidden layer of the image generation model. The noisy image is processed by the input layer and the hidden layer, and the output features of the hidden layer are fused with the text features corresponding to the description text through the cross-attention mechanism to guide the generation of the image through the description text.

[0049] When training the image generation model, high-quality images are used as sample images, and image descriptions of sample images are generated through a multimodal large language model. Entities and attributes in the image description are obtained, and the sample images are segmented based on the entity words and attributes to obtain the segmentation mask corresponding to each entity word attribute pair (including an entity word and an attribute). The image and image description after adding Gaussian noise to the sample image are input into the initial image generation model, and the noise distribution corresponding to the image description in the image after adding Gaussian noise is predicted through the initial image generation model. The network parameters of the image generation model are adjusted based on Gaussian noise, noise distribution and segmentation mask. During the model training process, the attention score of the cross-attention mechanism inside the segmentation mask is enhanced through the segmentation mask, and the attention score of the cross-attention mechanism outside the segmentation mask is reduced. This can guide the model to learn the image elements corresponding to entity words and attributes during the training process, and improve the semantic consistency between the generated image and the description text.

[0050] In an exemplary embodiment, the image generation model includes a diffusion network and a control network, wherein the diffusion network is used to control the picture quality of the image to be generated, and the control network is used to control the semantic consistency between the image to be generated and the description text;

[0051] When training the image generation model, the control network and the diffusion network are trained alternately, and the learning rate of the control network is greater than the learning rate of the diffusion network.

[0052] Among them, the control network, as a low-rank adaptation model, can fine-tune the diffusion network.

[0053] On the basis of training the diffusion network with a small learning rate, a control network based on LoRA (Low-Rank Adaptation, LoRA) is adopted to introduce new parameter variables, learn the semantic correlation between images and description texts with a larger learning rate, and improve the basic semantics of the image generation model, including color attribute changes, size changes, material relationships, spatial relationships, etc., so as to achieve the purpose of improving the aesthetics of the picture and semantic consistency at the same time.

[0054] Step 130: Generate a target image corresponding to the description text according to the noise image and the first predicted noise.

[0055] The first predicted noise is a noise distribution map of a size corresponding to the noise image. A difference image between the noise image and the first predicted noise can be calculated and used as a target image corresponding to the description text. Alternatively, the picture quality of the difference image can be further optimized to obtain a target image, so as to further improve the picture quality of the target image.

[0056] Taking an image design website as an example, after the image generation model training is completed, the description text entered by the user can be passed in through the front end. The image generation method provided in the embodiment of the present application can visualize the description text, generate a semantically consistent and beautiful target image, and return it to the front end through the interface.

[0057] The image generation method provided by the embodiment of the present application generates a noise image upon receiving a description text of an image to be generated, inputs the noise image and the description text into an image generation model, obtains a first predicted noise corresponding to the description text, and generates a target image corresponding to the description text based on the noise image and the first predicted noise. Since the image generation model learns the image elements corresponding to the entity words and attributes through the segmentation masks corresponding to the entity words and attributes during training, the semantic consistency and aesthetics of the generated target image and the description text can be improved, and there is no need to select the optimal image through retrieval or voting filtering, which reduces the computing cost and can improve the image generation efficiency. There is no need to preset an image library, which can save storage costs.

[0058] Figure 2 1 is a flow chart of another image generation method provided by an embodiment of the present application. Based on the above embodiment, this embodiment further refines the above step 130 to further optimize the image quality of the generated target image. Figure 2 As shown, the image generating method includes: step 210 to step 260.

[0059] Step 210: upon receiving a description text of an image to be generated, randomly generate a noise image.

[0060] Step 220: input the noise image and the description text into an image generation model to obtain a first predicted noise corresponding to the description text.

[0061] Among them, during training, the image generation model adjusts network parameters based on Gaussian noise added to the sample image, the predicted noise distribution and the segmentation mask, and the segmentation mask corresponds to the entity words and attributes in the image description corresponding to the sample image.

[0062] Step 230: Determine the difference between the noise image and the first predicted noise as the initial image corresponding to the description text.

[0063] The first predicted noise is subtracted from the noise image, and the difference obtained is used as the initial image corresponding to the description text. The picture quality of the initial image can be further optimized later to further improve the picture quality of the finally generated image.

[0064] Step 240: Add the image quality prompt words matching the description text to the description text to obtain a new description text.

[0065] The above process of generating an initial image based on text description is a process of generating an image from text. Next, the image quality of the initial image is optimized through an image generation model, which is a process of generating an image from an image.

[0066] You can search for image quality prompt words in a pre-set image quality prompt word library. Image quality prompt words can be, for example, "high image quality", "bright colors", "realistic", "textured", "movie texture", etc. Add the image quality prompt words that match the description text to the description text to obtain a new description text.

[0067] Step 250, input the initial image and the new description text into the image generation model, optimize the picture quality of the initial image according to the new description text through the image generation model, and obtain a second predicted noise corresponding to the initial image and the new description text.

[0068] The initial image and the new description text are input into the image generation model, the initial image is input into the input layer of the image generation model, the text features corresponding to the new description text are input into the hidden layer of the image generation model, the initial image is processed through the input layer and the hidden layer, the output features of the hidden layer are fused with the text features corresponding to the new description text through a cross-attention mechanism, so as to guide the image generation model to optimize the picture quality of the initial image through the new description text, and obtain a second predicted noise corresponding to the initial image and the new description text.

[0069] Step 260: Determine the difference between the initial image and the second predicted noise as the target image.

[0070] The second prediction noise is subtracted from the initial image, and the difference obtained is the target image corresponding to the description text.

[0071] It should be noted that the above-mentioned text-generated image stage needs to be denoised repeatedly for many times, and the image finally obtained is the initial image. The above-mentioned image-generated image stage also needs to be denoised repeatedly for many times, and finally the target image is obtained. The number of denoising in the image-generated image stage is less than the number of denoising in the text-generated image stage.

[0072] In the above-mentioned text generation image stage, Gaussian noise is randomly initialized to obtain a noise image, and the description text input by the user is text-featured and input together with the noise image into the image generation model based on the residual connection network, and the image is generated based on the initialized noise image minus the predicted first prediction noise. This process is repeated multiple times, for example, 30 times, to generate a high-quality initial image that meets the semantic description.

[0073] In the above-mentioned image generation stage, the initial image and the text features of the new description text are input into the image generation model, and the image quality is optimized by subtracting the predicted second prediction noise from the initial image. The entire denoising process is repeated multiple times, for example, 5 times, and the image details are fine-tuned to optimize the image quality without changing the image layout.

[0074] The image generation method provided in the embodiment of the present application generates an initial image based on a description text and a randomly generated noise image, and supplements the description text with image quality prompt words to obtain a new description text. The image quality of the initial image is optimized based on the new description text through an image generation model, which can further improve the aesthetics of the generated target image.

[0075] On the basis of the above technical solution, before the noise image and the description text are input into the image generation model, it also includes: determining a target main word that matches the words in the description text from multiple main words; superimposing the low-rank adaptation model corresponding to the target main word into the image generation model to obtain a new image generation model.

[0076] In the part-of-speech analysis stage, the main words of the description text input by the user are matched. If it matches the main words in the database, the main words are used as the target main words. At this time, the low-rank adaptation model corresponding to the target main word is loaded, and the low-rank adaptation model is superimposed with the basic model (i.e., the above-mentioned image generation model) to enhance the basic model's understanding of the specific input description. The description text obtained after the part-of-speech analysis is input into the new image generation model obtained after superimposing the low-rank adaptation model for image reasoning generation. Among them, the low-rank adaptation model, as a lightweight plug-in, can be superimposed with the basic model to fine-tune the basic model. LoRA is a lightweight model adjustment technology that achieves model fine-tuning by introducing low-rank decomposition in a pre-trained large model and adjusting only a small number of parameters. LoRA fine-tunes the model to adapt to new tasks or data sets without significantly increasing the computational burden by inserting a trainable low-rank decomposition matrix in certain layers of the pre-trained model.

[0077] By superimposing a low-rank adaptation model that matches the target subject words into the image generation model, the semantic consistency and aesthetics of the image finally generated by the image generation model and the description text can be further improved.

[0078] Figure 3 is a schematic diagram of the execution process of the image generation method provided in the embodiment of the present application, such as Figure 3 As shown in the figure, the whole process includes: model selection through part-of-speech analysis, receiving user input for text-generated image generation, obtaining the initial image, screening image quality prompt words, and optimizing the image-generated image in combination with the initial image. In the part-of-speech analysis stage, the main words of the description text entered by the user are matched. If it matches the target main words in the database, a lightweight low-rank adaptation model (Lora model) will be loaded and superimposed with the basic model (image generation model) to enhance the basic model's understanding of the specific input description. The image description obtained after the part-of-speech analysis is input into the current basic model for text-generated image reasoning generation. The generation process is basically the same as the training process, except that the sample image with Gaussian noise added during training is changed to random Gaussian noise for random initialization. The description text entered by the user is text-featured and input into the image generation model together with Gaussian noise for noise prediction, and the noise is subtracted from the Gaussian noise to generate the image. This process is repeated 30 times to generate a high-quality image that meets the semantic description. In the image generation stage, we randomly search for image quality prompt words in the pre-set image quality prompt word library, such as "high image quality", "bright colors", "realistic", and "textured", and add the image quality prompt words to the description text to construct a new description text, and replace the random Gaussian noise with the initial image obtained from the image generation model as the input of the image generation model for noise prediction. The entire denoising process is repeated 5 times, and the picture details are fine-tuned to optimize the picture quality without changing the picture layout.

[0079] Figure 4 1 is a flowchart of a method for training an image generation model provided in an embodiment of the present application. The method for training an image generation model is used to train the image generation model used in the above embodiment before sampling the image generation method provided in the above embodiment for image generation. Figure 4 As shown, the method may include steps 410 to 460.

[0080] Step 410: Generate an image description of the sample image.

[0081] The text of the image description of the sample image can be generated by a large language model.

[0082] In an exemplary embodiment, a large number of high-quality images may be collected as sample images, and image descriptions may be generated by using a multimodal large language model through prompt word engineering. The prompt words used may be as follows:

[0083] "1. The description should be no less than 50 words and no more than 500 words. 2. The first sentence of the description should begin with a description of the main subject of the image. 3. The description should be objective and avoid subjective emotions and speculation. 4. The description should include detailed information about the number, color, appearance and location of the items. 5. The description should follow a logical order, such as from left to right or from the center outward. 6. Avoid using prefixes such as "in the picture", "this image depicts" or "here is."

[0084] The process of generating image descriptions through a multimodal large language model can be expressed as follows:

[0085] Cap=MLLM(I,P pre )

[0086] Among them, P pre Represents the preset prompt word, I represents the sample image to be described, Cap represents the generated image description, and MLLM represents the multimodal large language model.

[0087] Step 420: Determine entity words in the image description and attributes corresponding to the entity words.

[0088] The part-of-speech parsing model can be used to perform syntactic parsing of image description. For example, the Spacy part-of-speech parsing model is used to parse entity words and adjectives in the sentence. Adjectives are attributes corresponding to entity words, and attributes and entities are bound through a syntactic tree structure to obtain entity pairs (n) and attribute-entity word binding pairs (an) in the description text.

[0089] Step 430 : Segment the sample image according to the entity word and the attribute, and obtain a segmentation mask corresponding to the entity word and the attribute.

[0090] A segmentation model can be used to segment the sample image. By inputting entity words and sample images into the segmentation model, the segmentation model outputs a corresponding segmentation mask, in which the segmented subject is white and the rest of the background is black. The segmentation model can be, for example, a Segment Anything Model (SAM).

[0091] The process of segmenting the sample image through the segmentation model can be expressed as follows:

[0092] M n =SAM(I,n)

[0093] Among them, I represents the sample image, n represents the entity word, and M n represents segmentation mask and SAM represents segmentation model.

[0094] The segmentation masks corresponding to the entity words can be screened based on the attributes corresponding to the entity words to determine the segmentation masks corresponding to both the entity words and the attributes, that is, to determine the segmentation masks corresponding to the entity word attribute pairs (including the entity words and the attributes).

[0095] In one embodiment of the present application, the sample image is segmented according to the entity word and the attribute to obtain a segmentation mask corresponding to the entity word and the attribute, including: inputting the entity word and the sample image into a segmentation model to obtain one or more initial segmentation masks corresponding to the entity word; for each of the attributes, selecting an initial segmentation mask that matches the attribute from the one or more initial segmentation masks as the segmentation mask corresponding to the entity word and the attribute.

[0096] The entity word and the sample image are input into the segmentation model, and one or more segmentation masks corresponding to the entity word are segmented from the sample image by the segmentation model as the initial segmentation mask of the entity word. When the same entity included in the sample image has multiple different attributes, for the same entity word, segmentation by the segmentation model can obtain the same number of segmentation masks as the attributes, and each segmentation mask corresponds to an attribute. However, the segmentation result of the segmentation model cannot distinguish the attributes. At this time, for each attribute, the initial segmentation mask matching the attribute can be screened out from the multiple initial segmentation masks as the segmentation mask corresponding to the entity word and the attribute (entity word attribute pair).

[0097] By segmenting the sample image based on the segmentation model and filtering the segmentation mask matching the attribute based on the attribute, the segmentation mask corresponding to the entity and the attribute can be accurately determined.

[0098] In one embodiment of the present application, for each of the attributes, selecting an initial segmentation mask matching the attribute from the one or more initial segmentation masks as a segmentation mask corresponding to the entity word and the attribute includes:

[0099] If the entity word corresponds to one of the attributes, an initial segmentation mask corresponding to the entity word is determined as a segmentation mask corresponding to the entity word and the attribute;

[0100] If the entity word corresponds to multiple attributes, the number of the initial segmentation masks is the same as the number of the attributes. According to the image features corresponding to each initial segmentation mask and the second text features corresponding to each entity word attribute pair, the segmentation mask corresponding to each entity word attribute pair is determined, and each entity word attribute pair includes the entity word and one of the attributes.

[0101] When the same entity word determined by the image description corresponds to an attribute, a corresponding segmentation mask can be obtained after segmenting the sample image through the segmentation model, and the segmentation mask can be directly determined as the segmentation mask corresponding to the entity word and the attribute.

[0102] When the same entity word determined by the image description corresponds to multiple attributes, a segmentation mask with the same number of attributes can be obtained after segmenting the sample image through the segmentation model. At this time, feature extraction can be performed on the area corresponding to each initial segmentation mask in the sample image to obtain image features corresponding to each initial segmentation mask, and the second text features corresponding to the entity word attribute pair can be extracted, and the image features and the second text features can be matched, and the initial segmentation mask with the highest matching degree with an entity word attribute pair can be determined as the segmentation mask corresponding to the entity word attribute pair.

[0103] In an exemplary embodiment, the segmentation mask corresponding to each entity word attribute pair is determined based on the image feature corresponding to each initial segmentation mask and the second text feature corresponding to each entity word attribute pair, including: for each entity word attribute pair, determining the similarity between the second text feature corresponding to the entity word attribute pair and the image feature corresponding to each initial segmentation mask; and determining the initial segmentation mask with the largest similarity as the segmentation mask corresponding to the entity word attribute pair.

[0104] For example, since the segmentation model cannot accurately distinguish attribute concepts, and when there are multiple entities with the same but different attributes in the image, the corresponding entities cannot be accurately distinguished by the segmentation model alone, the multimodal perception model CLIP (Contrastive Language-Image Pre-training) can be used as an expert network to determine the relevance between the segmented area and the corresponding attribute-entity word, and further filter according to the relevance:

[0105] Emb i =E i (I n ),

[0106] Emb t =E t (a+n),

[0107] sim=cosine(Emb i ,Emb t ).

[0108] Among them, I n represents the area corresponding to entity word n ​​in sample image I, E i (I n ) represents the image features of the area corresponding to the i-th initial segmentation mask corresponding to the entity word n ​​in the sample image I, Emb i represents the image feature corresponding to the i-th initial segmentation mask, a represents the attribute, n represents the entity word, E t (a+n) means finding the second text feature corresponding to entity word n ​​and attribute a. t It represents the second text feature corresponding to the entity word attribute pair (an), and sim represents the similarity between the image feature and the second text feature.

[0109] If there are multiple identical entities with different attributes in the sample image, the segmentation mask corresponding to the entity word attribute pair is determined by similarity. The accuracy of the correspondence between the determined entity word attribute pair and the segmentation mask can be improved, thereby improving the semantic consistency between the image generated by the trained model and the description text.

[0110] Through the above method, for a sample image, the corresponding relationship between the image description, segmentation mask, attribute, and entity corresponding to the sample image can be obtained, that is, the sample image-image description-segmentation mask-attribute-entity pair (I, Cap, M n ,a,n), I represents the sample image, Cap represents the image description, M n represents the segmentation mask, a represents the attribute, and n represents the entity word.

[0111] Figure 5 Schematic diagram of the process of constructing training data in the embodiment of the present application. Figure 5 As shown in Figure 1, the process of constructing training data includes: for a sample image, firstly, a multimodal large language model is used to generate an image description of the sample image, and then Figure 5 Taking the sample image shown as an example, the obtained image description is "a skateboard, a tennis racket and a yellow cup"; the entity decomposition analysis is performed through the part-of-speech analysis model, and the entity words obtained are "skateboard", "tennis racket", and "cup"; the attribute decomposition analysis is performed through the part-of-speech analysis model, and the entity word attribute pairs are "skateboard", "tennis racket", and "yellow cup"; the segmentation mask is detected for each entity word attribute pair, and the mask label is constructed according to the detection result, that is, the segmentation mask corresponding to each entity word attribute pair is given.

[0112] Step 440: superimpose Gaussian noise onto the sample image to obtain a noise image.

[0113] By randomly sampling a timestamp t, Gaussian noise is gradually superimposed on the sample image z0(I) to obtain a noise image, which can be expressed as the following formula:

[0114]

[0115] Among them, ε represents Gaussian noise, and α t represents the coefficient that determines the degree of noise increase, z0(I) represents the sample image, and z t (I) represents a noisy image.

[0116] Step 450, input the noise image and the image description into an initial image generation model to obtain a noise distribution corresponding to the noise image and the image description; in the image generation model, the first text feature of the image description and the hidden layer image feature are fused through a cross-attention mechanism.

[0117] Among them, the image generation model includes a diffusion network and a control network, the diffusion network is used to control the picture quality of the image to be generated, and the control network is used to control the semantic consistency between the image to be generated and the image description; the control network and the diffusion network are trained alternately, and the learning rate of the control network is greater than the learning rate of the diffusion network.

[0118] The constructed sample image-image description-segmentation mask-attribute-entity pair (I, Cap, M n , a, n) as the input of the image generation model, and the input is preprocessed. For text, the multimodal perception model CLIP is used to model text features, which is expressed as follows:

[0119] E t = CLIP(Tokenize(Cap))

[0120] Among them, Tokenize means tokenizing (segmenting and encoding) the description text Cap, obtaining the token (encoding) of the corresponding word, and inputting it into CLIP to obtain the feature vector of the corresponding word token. The feature vectors of all words in the description text are combined into the feature vector E describing the text. t .

[0121] In order to achieve image reconstruction, a diffusion network (Unet network) is used to model the noisy image z t (I) to learn the noise distribution ε superimposed on the sample image z0(I) θ , and the noise image z t(I) subtract to reconstruct the original image (i.e., the sample image). By learning different noise distributions, the diffusion network can reconstruct and generate different high-quality images. In order to achieve text-guided image generation, the text feature E t It is injected into the diffusion network and fused with the intermediate features of the image in the diffusion network (hidden layer image features) through cross-attention connection. The fusion formula is as follows:

[0122]

[0123] Among them, U i is the hidden layer feature in the diffusion network, W represents the transformation matrix, E t represents the first text feature, U i+1 represents the image features after text information injection, that is, the fused image features, and d represents the number of multi-head attention heads in the cross-attention mechanism.

[0124] When training the model, when the learning rate is large, the generative network (image generation model) is difficult to retain the original image generation aesthetics, and when the learning rate is small, the semantics cannot be effectively improved. To this end, on the basis of training the diffusion network with a small learning rate, the Lora control network is used to introduce new parameter variables, learn the semantic relevance in the image and text with a larger learning rate, and improve the basic semantics of the model, including color attribute changes, size changes, material relationships, spatial relationships, etc. Specifically, the control network adds a bypass branch network on the basis of the existing diffusion network, and introduces new parameters by reducing the dimension of the input features (MLP matrix A) and increasing the dimension (MLP matrix B). In actual training, matrix A is initialized with a random Gaussian distribution, and B is initialized with a 0 matrix, so as to gradually introduce new semantic information.

[0125] The combination of diffusion network and control network can be expressed as follows:

[0126] W+ΔW=W+BA

[0127] h=W+ΔWx=Wx+BAx=(W+BA)x.

[0128] Where W represents the diffusion network, B and A are the newly added control networks, x represents the hidden layer input features, and h represents the hidden layer output features. During the training of the control network, the diffusion network parameters will be fixed to learn the semantic relevance between the constructed image-text pairs.

[0129] In order to improve the aesthetics and semantic consistency of the picture at the same time, the existing network and the control network are decoupled for alternate training, so that the entire image generation model can focus on one goal for training at the same time, which can greatly reduce the difficulty of training and accelerate model convergence.

[0130] Step 460, adjusting the network parameters of the initial image generation model according to the Gaussian noise, the noise distribution and the segmentation mask to obtain a trained image generation model.

[0131] Based on the Gaussian noise added to the sample image, the predicted noise distribution and the segmentation mask, the network parameters of the initial image generation model are adjusted, and the sample image and image description are input, and the network parameters of the image generation model are adjusted iteratively until the training end condition is met, and the trained image generation model is obtained. The training end condition can be that the number of iterations reaches a preset number, the network parameters converge, or the loss function value converges.

[0132] When the diffusion network and the control network are alternately trained, after the noise image and image description corresponding to a sample image are input into the image generation model, the parameters of the control network can be fixed first, and the network parameters of the diffusion network can be adjusted based on the noise distribution, Gaussian noise and segmentation mask output by the image generation model. Then, the network parameters of the diffusion network are fixed, and the noise image and image description corresponding to the same sample image are input into the image generation model again. Based on the noise distribution, Gaussian noise and segmentation mask output by the image generation model, the network parameters of the control network are adjusted. In this way, one round of training is completed, and then other sample images are used for the next round of alternating training.

[0133] In one embodiment of the present application, adjusting the network parameters of the initial image generation model according to the Gaussian noise, the noise distribution and the segmentation mask to obtain the trained image generation model includes:

[0134] Determining a mean square error loss function value of the image generation model according to the Gaussian noise and the noise distribution;

[0135] Determine a mask optimization loss function value according to the segmentation mask, where the mask optimization loss function is used to enhance a first attention score of the cross-attention mechanism inside the segmentation mask and reduce a second attention score of the cross-attention mechanism outside the segmentation mask;

[0136] According to the mean square error loss function value and the mask optimization loss function value, the network parameters of the initial image generation model are adjusted to obtain a trained image generation model.

[0137] According to the Gaussian noise and the predicted noise peak, the mean square error loss function value of the image generation model is determined according to the following formula:

[0138]

[0139] Among them, L ldm represents the mean square error loss function value, ∈ represents Gaussian noise, z t (I) represents the noise image, t represents the timestamp, E t represents the first text feature, ε θ (z t (I),t,E t ) represents the noise distribution predicted by the image generation model.

[0140] The image generation model can be trained based on the mean square error loss function to maintain the original picture quality and gradually migrate to a new picture style.

[0141] The joint optimization of some parameters of the image generation model is achieved through the mean square error loss function. Figure 6 is a schematic diagram of the process of joint optimization of some parameters in the embodiment of the present application, such as Figure 6 As shown, an autoregressive generative network VAE is used to encode the sample image, and Gaussian noise is superimposed on the encoded sample image to obtain a noise image. The noise image is then input into the diffusion network (Unet network) of the image generation model, and processed through each network layer of the diffusion network. The image description is input into the hidden layer and control network of the diffusion network. The diffusion network fuses the first text features of the image description and the hidden layer image features through a cross-attention mechanism, and is combined with the control network to jointly process the hidden layer image features to obtain a predicted noise distribution. Based on the noise distribution and the Gaussian noise added to the sample image, the mean square error loss function value is determined, and the network parameters of the image generation model are adjusted. For the alternating training of the diffusion network and the control network in the image generation model, the specific alternating training method can refer to the description of the above embodiment, which will not be repeated here.

[0142] Figure 7 Schematic diagram of mask loss optimization in an embodiment of the present application. Figure 7 As shown, in order to improve semantic consistency, the embodiment of the present application proposes a mask loss optimization process at the entity word level, calculates the loss of hidden layer features and semantic segmentation features, enhances the understanding and generation of spatial layout attributes of entity words, and through feature masking and segmentation feature supervision, enhances the attention weight in semantically related areas and reduces information interference in semantically irrelevant areas.

[0143] The first text feature is fused with the hidden layer image feature at each layer through the cross-attention mechanism, so that the image generation model can be guided by the text information. In the process of constructing the cross-attention mechanism, the attention score of the cross-attention mechanism intuitively expresses the importance of the corresponding first text feature in a certain area. The attention score of the cross-attention mechanism is expressed as follows:

[0144]

[0145] Among them, A u Represents the attention score, which is an m*n two-dimensional matrix, where n represents the number of text tokens (i.e. entity words) and m is the product of the length and width of the image hidden layer feature. u The value in the i-th row and j-th column of represents the probability of the i-th text token appearing at the j-th position. Therefore, by constraining a specific token to appear at a specific position and suppressing its appearance in irrelevant areas, that is, enhancing the first attention score of the cross-attention mechanism inside the segmentation mask and reducing the second attention score of the cross-attention mechanism outside the segmentation mask, semantic consistency can be effectively improved.

[0146] Finally, the overall loss function of the image generation model can be expressed as follows:

[0147] L total =L ldm +L token

[0148] Among them, L total Represents the overall loss function value, L ldm Represents the mean square error loss function value, L token Represents the mask optimization loss function value.

[0149] In one embodiment of the present application, determining the mask optimization loss function value based on the segmentation mask includes: determining the mask internal loss function value regarding the first attention score based on the segmentation mask, and determining the mask external loss function value regarding the second attention score; and determining the sum of the mask internal loss function value and the mask external loss function value as the mask optimization loss function value.

[0150] The calculation formulas of the mask internal loss function value and the mask external loss function value are as follows:

[0151]

[0152] in, represents the internal loss function value of the mask, Represents the mask external loss function value, A u represents the attention score of the cross-attention mechanism, M represents the segmentation mask, and A u *M represents the first attention score, A u *(1-M) represents the second attention score. u is the matrix of overall attention scores.

[0153] The image-image description-segmentation mask-attribute-entity pair (I, Cap, M n ,a,n) as fine-grained labels. By enhancing the corresponding regions of attributes and entities, u , and reduce the A corresponding to the irrelevant background u , as an additional optimization target to assist model training.

[0154] In the calculation formulas of the above-mentioned mask internal loss function value and the mask external loss function value, the mask internal loss function is used to enhance the first cross-attention score inside the mask, where M is the segmentation mask corresponding to the entity and attribute, 1 is the specified area in the segmentation mask, and 0 is the irrelevant background area. By calculating max(A u *M) to get the maximum value of the cross-attention score within the mask and bring it closer to 1 to increase the probability of the attribute entity appearing in the mask. Mask external loss function It is used to reduce the cross-attention score outside the mask, and suppress the overall distribution by reducing the maximum value of the cross-attention score outside the mask. The final overall mask optimization loss function is as follows:

[0155]

[0156] Among them, L token represents the mask optimization loss function.

[0157] By introducing the mask optimization loss function, the hidden layer image representation distribution at the token (entity word) level is redistributed to enhance the probability of each entity appearing at a specific location. At the same time, the attribute corresponding mask used in the embodiment of the present application is the same as the entity mask corresponding to the attribute, which materializes the attribute representation, greatly enhances the accuracy of attribute binding, and improves semantic consistency.

[0158] The training method of the image generation model provided in the embodiment of the present application generates an image description of a sample image, determines the entity words in the image description and the attributes corresponding to the entity words, segments the sample image according to the entity words and the attributes, obtains the segmentation mask corresponding to the entity words and the attributes, superimposes Gaussian noise on the sample image to obtain a noise image, inputs the noise image and the image description into the initial image generation model, obtains the noise distribution corresponding to the noise image and the image description, and adjusts the network parameters of the initial image generation model according to the Gaussian noise, the noise distribution and the segmentation mask to obtain the trained image generation model. Since the model learns the image elements corresponding to the entity words and the attributes through the segmentation mask corresponding to the entity words and the attributes, the semantic consistency and the aesthetics of the picture between the generated target image and the description text can be improved, and there is no need to select the optimal image by retrieval or voting filtering, which reduces the computing cost, can improve the image generation efficiency, and does not require a preset image library, which can save storage costs.

[0159] On the basis of the above technical solution, before superimposing Gaussian noise on the sample image to obtain the noise image, the method further includes: encoding the sample image to compress the sample image into a sample image of a preset size.

[0160] Before using sample images to train the image generation model, the sample images are preprocessed. For images, in order to reduce the video memory usage of the image generation model during training, an autoregressive generative network VAE (Variational Auto-Encoder) can be used to encode the sample images, compress the sample images to a preset size, and use the compressed sample images as z0(I), and Gaussian noise can be superimposed later. The preset size can be, for example, 128*128*3, and the original size of the sample image can be 1024*1204*3.

[0161] In summary, the embodiments of the present application can be used for text generation image tasks for creative image generation and design. The process of model training and application mainly includes: 1. Describing the sample image in natural language, detecting and segmenting the main area in the sample image, and binding and aligning it with the natural language to build a fine-grained training data set. 2. Building a training method for joint alternating optimization of some parameters to maintain the existing picture quality during model training. 3. Optimizing the mask loss function, supervising the text-image interaction middle layer (hidden layer), maximizing the cross-attention distribution of attribute entities in the matching mask, minimizing the attention distribution outside the mask, encouraging the model to generate specified entities within the mask, and reducing semantic confusion and attention leakage. 4. Using the trained image generation model as the basic model, receiving the text description as input, and quickly generating a semantically consistent target image. The embodiment of the present application effectively improves the semantic consistency of the text-image model, reduces the model training time, and reduces costs by constructing a training method for multimodal block reconstruction.

[0162] The embodiments of the present application effectively solve the problem of lack of training data for text graphs, provide a general fine-grained image-text pair generation method, and generate image-image description-segmentation mask-attribute-entity pairs; propose a semantic consistency alignment training method for block reconstruction, and train the model to achieve more accurate image generation under limited data conditions, effectively solving the problem of mismatch between images generated by the text graph model and user input; propose semantic matching generation based on text graphs, and image quality optimization of text graphs, to solve the problem of insufficient aesthetics of the text graph model.

[0163] Figure 8 is a schematic diagram of the structure of an image generating device provided in an embodiment of the present application, such as Figure 8 As shown, the image generating device comprises:

[0164] A noise image generating module 810, configured to randomly generate a noise image upon receiving a description text of an image to be generated;

[0165] The noise prediction module 820 is used to input the noise image and the description text into an image generation model to obtain a first predicted noise corresponding to the description text; the image generation model is trained by adjusting network parameters based on the Gaussian noise added to the sample image, the predicted noise distribution and the segmentation mask, and the segmentation mask corresponds to the entity words and attributes in the image description corresponding to the sample image;

[0166] The image generation module 830 is used to generate a target image corresponding to the description text according to the noise image and the first predicted noise.

[0167] Optionally, the image generation module includes:

[0168] an initial image determining unit, configured to determine a difference between the noise image and the first predicted noise as an initial image corresponding to the description text;

[0169] A text updating unit, used for adding a picture quality prompt word matching the description text to the description text to obtain a new description text;

[0170] an image quality optimization unit, configured to input the initial image and the new description text into the image generation model, optimize the image quality of the initial image according to the new description text through the image generation model, and obtain a second predicted noise corresponding to the initial image and the new description text;

[0171] A target image determining unit is used to determine the difference between the initial image and the second predicted noise as the target image.

[0172] Optionally, the image generation model includes a diffusion network and a control network, the diffusion network is used to control the picture quality of the image to be generated, and the control network is used to control the semantic consistency between the image to be generated and the description text;

[0173] When training the image generation model, the control network and the diffusion network are trained alternately, and the learning rate of the control network is greater than the learning rate of the diffusion network.

[0174] Optionally, the device further comprises:

[0175] A subject word matching module, used to determine a target subject word that matches a word in the description text from a plurality of subject words;

[0176] The model updating module is used to superimpose the low-rank adaptation model corresponding to the target subject word into the image generation model to obtain a new image generation model.

[0177] The image generation device provided in the embodiment of the present application is used to implement the various steps of the image generation method described in the embodiment of the present application. The specific implementation methods of each module of the device can be found in the corresponding steps and will not be repeated here.

[0178] The image generation device provided by the embodiment of the present application generates a noise image upon receiving a description text of an image to be generated, inputs the noise image and the description text into an image generation model, obtains a first predicted noise corresponding to the description text, and generates a target image corresponding to the description text based on the noise image and the first predicted noise. Since the image generation model learns the image elements corresponding to the entity words and attributes through the segmentation masks corresponding to the entity words and attributes during training, the semantic consistency and aesthetics of the generated target image and the description text can be improved, and there is no need to select the optimal image through retrieval or voting filtering, which reduces the computing cost and can improve the image generation efficiency. There is no need to preset an image library, which can save storage costs.

[0179] Fig. 9 is a structural diagram of a training device for an image generation model provided in an embodiment of the present application, such as Fig. 9 As shown, the training device of the image generation model includes:

[0180] An image description generating module 910, used to generate an image description of a sample image;

[0181] An entity word attribute determination module 920 is configured to determine entity words in the image description and attributes corresponding to the entity words;

[0182] An image segmentation module 930 is used to segment the sample image according to the entity word and the attribute, and obtain a segmentation mask corresponding to the entity word and the attribute;

[0183] A noise superposition module 940 is used to superimpose Gaussian noise onto the sample image to obtain a noise image;

[0184] A noise distribution prediction module 950 is used to input the noise image and the image description into an initial image generation model to obtain a noise distribution corresponding to the noise image and the image description; and to fuse the first text feature of the image description with the hidden layer image feature in the image generation model through a cross-attention mechanism;

[0185] The parameter adjustment module 960 is used to adjust the network parameters of the initial image generation model according to the Gaussian noise, the noise distribution and the segmentation mask to obtain a trained image generation model.

[0186] Optionally, the parameter adjustment module includes:

[0187] A first loss determination unit, configured to determine a mean square error loss function value of the image generation model according to the Gaussian noise and the noise distribution;

[0188] A second loss determination unit, configured to determine a mask optimization loss function value according to the segmentation mask, wherein the mask optimization loss function is used to enhance a first attention score of the cross-attention mechanism inside the segmentation mask and to reduce a second attention score of the cross-attention mechanism outside the segmentation mask;

[0189] A parameter adjustment unit is used to adjust the network parameters of the initial image generation model according to the mean square error loss function value and the mask optimization loss function value to obtain a trained image generation model.

[0190] Optionally, the second loss determination unit is specifically configured to:

[0191] Determine, based on the segmentation mask, a mask internal loss function value for the first attention score, and determine a mask external loss function value for the second attention score;

[0192] The sum of the mask internal loss function value and the mask external loss function value is determined as the mask optimization loss function value.

[0193] Optionally, the calculation formulas for the mask internal loss function value and the mask external loss function value are as follows:

[0194]

[0195]

[0196] in, represents the internal loss function value of the mask, Represents the mask external loss function value, A u represents the attention score of the cross-attention mechanism, M represents the segmentation mask, and A u *M represents the first attention score, A u *(1-M) represents the second attention score.

[0197] Optionally, the image segmentation module includes:

[0198] An image segmentation unit, used for inputting the entity word and the sample image into a segmentation model to obtain one or more initial segmentation masks corresponding to the entity word;

[0199] The matching unit is used to select, for each of the attributes, an initial segmentation mask that matches the attribute from the one or more initial segmentation masks as a segmentation mask corresponding to the entity word and the attribute.

[0200] Optionally, the matching unit includes:

[0201] A first matching subunit is used for determining an initial segmentation mask corresponding to the entity word as a segmentation mask corresponding to the entity word and the attribute if the entity word corresponds to one of the attributes;

[0202] The second matching subunit is used to determine the segmentation mask corresponding to each entity word attribute pair according to the image features corresponding to each initial segmentation mask and the second text features corresponding to each entity word attribute pair if the entity word corresponds to multiple attributes and the number of the initial segmentation masks is the same as the number of the attributes, wherein each entity word attribute pair includes the entity word and one of the attributes.

[0203] Optionally, the second matching subunit is specifically used for:

[0204] For each of the entity word attribute pairs, determining the similarity between the second text feature corresponding to the entity word attribute pair and the image feature corresponding to each initial segmentation mask;

[0205] The initial segmentation mask with the greatest similarity is determined as the segmentation mask corresponding to the entity word attribute pair.

[0206] Optionally, the device further comprises:

[0207] The image encoding module is used to encode the sample image and compress the sample image into a sample image of a preset size.

[0208] Optionally, the image generation model includes a diffusion network and a control network, the diffusion network is used to control the picture quality of the image to be generated, and the control network is used to control the semantic consistency between the image to be generated and the image description;

[0209] The control network and the diffusion network are trained alternately, and the learning rate of the control network is greater than the learning rate of the diffusion network.

[0210] The image generation model training device provided in the embodiment of the present application is used to implement the various steps of the image generation model training method described in the embodiment of the present application. The specific implementation methods of each module of the device refer to the corresponding steps and will not be repeated here.

[0211] The training device of the image generation model provided in the embodiment of the present application generates an image description of a sample image, determines the entity words in the image description and the attributes corresponding to the entity words, segments the sample image according to the entity words and the attributes, obtains the segmentation mask corresponding to the entity words and the attributes, superimposes Gaussian noise on the sample image to obtain a noise image, inputs the noise image and the image description into the initial image generation model, obtains the noise distribution corresponding to the noise image and the image description, and adjusts the network parameters of the initial image generation model according to the Gaussian noise, the noise distribution and the segmentation mask to obtain the trained image generation model. Since the model learns the image elements corresponding to the entity words and the attributes through the segmentation mask corresponding to the entity words and the attributes, the semantic consistency and the aesthetics of the picture between the generated target image and the description text can be improved, and there is no need to select the optimal image through a retrieval method or a voting filtering method, which reduces the computing cost, can improve the image generation efficiency, and does not require a preset image library, which can save storage costs.

[0212] Fig.10 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application, such as Fig.10 As shown, the electronic device 1000 may include one or more processors 1010 and one or more memories 1020 connected to the processor 1010. The electronic device 1000 may also include an input interface 1030 and an output interface 1040 for communicating with another device or system. The program code executed by the processor 1010 may be stored in the memory 1020.

[0213] The processor 1010 in the electronic device 1000 calls the program code stored in the memory 1020 to execute the image generation method or the image generation model training method in the above-mentioned embodiments.

[0214] An embodiment of the present application also provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps of the image generation method or the image generation model training method described in the embodiment of the present application are implemented.

[0215] An embodiment of the present application also provides a computer program product, which, when executed by a processor, implements the steps of the image generation method or the image generation model training method as described in the embodiment of the present application.

[0216] Each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0217] The above is a detailed introduction to an image generation method, model training method, device, equipment and storage medium provided in the embodiments of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the ideas of the present application, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on the present application.

[0218] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

Claims

1. An image generation method, characterized in that: include: When receiving a description text of an image to be generated, randomly generating a noise image; Inputting the noise image and the description text into an image generation model to obtain a first predicted noise corresponding to the description text; during training, the image generation model adjusts network parameters based on the Gaussian noise added to the sample image, the predicted noise distribution, and the segmentation mask, wherein the segmentation mask corresponds to entity words and attributes in the image description corresponding to the sample image; A target image corresponding to the description text is generated according to the noise image and the first predicted noise.

2. The method according to claim 1, characterized in that The step of generating a target image corresponding to the description text according to the noise image and the predicted noise includes: Determine the difference between the noise image and the first predicted noise as the initial image corresponding to the description text; Adding the image quality prompt words matching the description text to the description text to obtain a new description text; Inputting the initial image and the new description text into the image generation model, optimizing the picture quality of the initial image according to the new description text through the image generation model, and obtaining a second predicted noise corresponding to the initial image and the new description text; The difference between the initial image and the second predicted noise is determined as the target image.

3. The method according to claim 1 or 2, characterized in that: The image generation model includes a diffusion network and a control network, wherein the diffusion network is used to control the picture quality of the image to be generated, and the control network is used to control the semantic consistency between the image to be generated and the description text; When training the image generation model, the control network and the diffusion network are trained alternately, and the learning rate of the control network is greater than the learning rate of the diffusion network.

4. The method according to claim 1 or 2, characterized in that: Before inputting the noise image and the description text into the image generation model, the method further includes: Determining a target subject word that matches a word in the description text from a plurality of subject words; The low-rank adaptation model corresponding to the target subject word is superimposed on the image generation model to obtain a new image generation model.

5. A training method for an image generation model, characterized in that: include: Generate image descriptions of sample images; Determining entity words in the image description and attributes corresponding to the entity words; Segmenting the sample image according to the entity word and the attribute to obtain a segmentation mask corresponding to the entity word and the attribute; Superimposing Gaussian noise onto the sample image to obtain a noise image; Inputting the noise image and the image description into an initial image generation model to obtain a noise distribution corresponding to the noise image and the image description; fusing the first text feature of the image description with the hidden layer image feature in the image generation model through a cross-attention mechanism; According to the Gaussian noise, the noise distribution and the segmentation mask, the network parameters of the initial image generation model are adjusted to obtain a trained image generation model.

6. The method according to claim 5, characterized in that The adjusting the network parameters of the initial image generation model according to the Gaussian noise, the noise distribution and the segmentation mask to obtain the trained image generation model includes: Determining a mean square error loss function value of the image generation model according to the Gaussian noise and the noise distribution; Determine a mask optimization loss function value according to the segmentation mask, where the mask optimization loss function is used to enhance a first attention score of the cross-attention mechanism inside the segmentation mask and reduce a second attention score of the cross-attention mechanism outside the segmentation mask; According to the mean square error loss function value and the mask optimization loss function value, the network parameters of the initial image generation model are adjusted to obtain a trained image generation model.

7. The method according to claim 6, characterized in that Determining the mask optimization loss function value according to the segmentation mask includes: Determine, based on the segmentation mask, a mask internal loss function value for the first attention score, and determine a mask external loss function value for the second attention score; The sum of the mask internal loss function value and the mask external loss function value is determined as the mask optimization loss function value.

8. The method according to claim 7, characterized in that The calculation formulas of the mask internal loss function value and the mask external loss function value are as follows: in, represents the internal loss function value of the mask, Represents the mask external loss function value, A u represents the attention score of the cross-attention mechanism, M represents the segmentation mask, and A u *M represents the first attention score, A u *(1-M) represents the second attention score.

9. The method according to any one of claims 5 to 8, characterized in that: The step of segmenting the sample image according to the entity word and the attribute to obtain a segmentation mask corresponding to the entity word and the attribute includes: Inputting the entity word and the sample image into a segmentation model to obtain one or more initial segmentation masks corresponding to the entity word; For each of the attributes, an initial segmentation mask matching the attribute is selected from the one or more initial segmentation masks as a segmentation mask corresponding to the entity word and the attribute.

10. The method according to claim 9, characterized in that For each of the attributes, selecting an initial segmentation mask matching the attribute from the one or more initial segmentation masks as a segmentation mask corresponding to the entity word and the attribute includes: If the entity word corresponds to one of the attributes, an initial segmentation mask corresponding to the entity word is determined as a segmentation mask corresponding to the entity word and the attribute; If the entity word corresponds to multiple attributes, the number of the initial segmentation masks is the same as the number of the attributes. According to the image features corresponding to each initial segmentation mask and the second text features corresponding to each entity word attribute pair, the segmentation mask corresponding to each entity word attribute pair is determined, and each entity word attribute pair includes the entity word and one of the attributes.

11. The method according to claim 10, characterized in that The step of determining a segmentation mask corresponding to each entity word attribute pair according to the image feature corresponding to each initial segmentation mask and the second text feature corresponding to each entity word attribute pair comprises: For each of the entity word attribute pairs, determining the similarity between the second text feature corresponding to the entity word attribute pair and the image feature corresponding to each initial segmentation mask; The initial segmentation mask with the greatest similarity is determined as the segmentation mask corresponding to the entity word attribute pair.

12. The method according to any one of claims 5 to 8, characterized in that: Before superimposing Gaussian noise onto the sample image to obtain a noise image, the method further includes: The sample image is encoded and compressed into a sample image of a preset size.

13. The method according to any one of claims 5 to 8, characterized in that: The image generation model includes a diffusion network and a control network, wherein the diffusion network is used to control the picture quality of the image to be generated, and the control network is used to control the semantic consistency between the image to be generated and the image description; The control network and the diffusion network are trained alternately, and the learning rate of the control network is greater than the learning rate of the diffusion network.

14. An image generating device, characterized in that: include: A noise image generation module, used for randomly generating a noise image when receiving a description text of an image to be generated; A noise prediction module, used for inputting the noise image and the description text into an image generation model to obtain a first predicted noise corresponding to the description text; the image generation model is trained by adjusting network parameters based on Gaussian noise added to the sample image, the predicted noise distribution and the segmentation mask, and the segmentation mask corresponds to the entity words and attributes in the image description corresponding to the sample image; An image generation module is used to generate a target image corresponding to the description text according to the noise image and the first predicted noise.

15. A training device for an image generation model, characterized in that: include: An image description generation module, used to generate image descriptions of sample images; An entity word attribute determination module, used to determine the entity words in the image description and the attributes corresponding to the entity words; An image segmentation module, used to segment the sample image according to the entity word and the attribute, and obtain a segmentation mask corresponding to the entity word and the attribute; A noise superposition module, used for superimposing Gaussian noise onto the sample image to obtain a noise image; A noise distribution prediction module is used to input the noise image and the image description into an initial image generation model to obtain a noise distribution corresponding to the noise image and the image description; and to fuse the first text feature of the image description with the hidden layer image feature in the image generation model through a cross-attention mechanism; A parameter adjustment module is used to adjust the network parameters of the initial image generation model according to the Gaussian noise, the noise distribution and the segmentation mask to obtain a trained image generation model.

16. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, it implements the image generation method described in any one of claims 1 to 4 or the image generation model training method described in any one of claims 5 to 13.

17. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, it implements the steps of the image generation method described in any one of claims 1 to 4 or the steps of the image generation model training method described in any one of claims 5 to 13.

Images