A method for generating images from text

Abstract

The invention discloses a method for generating an image from text, and relates to the field of image generation. The method comprises: acquiring multi-entity sample data, and obtaining a recognition model based on the multi-entity sample data; recognizing the multi-entity sample data based on the recognition model to obtain a plurality of first entities, encoding each first entity to obtain a plurality of single vectors, and fusing all the single vectors to obtain a comprehensive vector; obtaining a second image based on the comprehensive vector and a generator, obtaining a target generation parameter of the generator based on the second image, and obtaining a generation model based on the target generation parameter; acquiring text data, recognizing the text data based on the recognition model to obtain a plurality of second entities, and using the generation model to obtain images corresponding to the text data based on all the second entities, and can solve the problem that the existing cross-modal learning method lacks structured processing of text and images, resulting in increased information loss in cross-modal learning, thereby resulting in reduced diversity and authenticity of generated images.

Description

Technical Field

[0001] The present invention relates to the field of image generation, and in particular to a method for generating images from texts. Background Art

[0002] With the rapid development of science and technology, the data types on the Internet have become more diverse, including but not limited to text, images, voice, and video. When processing different types of data, converting data of a certain modality into another data type can effectively break the feature description barriers between different types of data. Currently, cross-modal learning methods are commonly used for mutual conversion, but this method lacks structured processing of unstructured data such as text and images, resulting in increased information loss in cross-modal learning, which leads to reduced diversity and authenticity of generated images. Summary of the invention

[0003] In order to solve the problem that existing cross-modal learning methods lack structured processing of text and images, resulting in increased information loss in cross-modal learning, the present invention provides a method for generating images from text, the method comprising: acquiring single-entity sample data, obtaining multi-entity sample data based on the single-entity sample data, and obtaining a recognition model based on the multi-entity sample data; recognizing the multi-entity sample data based on the recognition model to obtain a number of first entities, encoding each of the first entities to obtain a number of single vectors, and fusing all the single vectors to obtain a comprehensive vector; obtaining a second image based on the comprehensive vector and a generator, and obtaining a first loss L of the generator based on the second image. G , based on the first loss L G Obtain target generation parameters of the generator, and obtain a generation model based on the target generation parameters; obtain text data, recognize the text data based on the recognition model to obtain a number of second entities, and the generation model obtains images corresponding to the text data based on all the second entities.

[0004] The principle of the present invention is as follows: by splicing single entity sample data to obtain multi-entity sample data, the amount of sample data is increased, so that the recognition model and the generation model can be trained more comprehensively and accurately; all entities are encoded and embedded to obtain a comprehensive vector, and the unstructured data of the text data is converted into structured data. The vector captures the joint semantic information of all entity labels, and then forms the input condition of the generator with a set of random noise vectors. The condition directly affects the features of different resolutions and hierarchical levels of the generated image, and guides the generator to generate images that reflect the semantic information of the entity labels and have diversity; then the structured judgment of the generated image is introduced by calculating the information loss of the generator. In the first stage, the loss of the discriminator is calculated and returned to the generator. The generation parameters of the generator are continuously optimized through the loss, so that the optimal The image generated by the optimized generation parameters can be judged as a real image by the discriminator, which can reduce the information loss on unstructured structured data such as text and images, and continuously optimize the generation parameters of the generator by calculating the loss, so as to continuously adjust and optimize the generated image, improve the image quality and improve the diversity and authenticity of the generated image; in the second stage, the loss of the entity type and number of the text and the generated image is calculated, and it is returned to the generator. The generation parameters of the generator are continuously optimized again through the loss until the entities and quantities corresponding to the text are generated, and the final target generation parameters of the generator are obtained. The generated image is optimized again to improve the accuracy, diversity and authenticity of the generated image. The model is trained by the above training method to obtain the generation model, so that the generation model can generate corresponding images according to the entities of the text.

[0005] Furthermore, the specific steps of obtaining multi-entity sample data based on the single entity sample data include: the single entity sample data includes a plurality of first images and first text information of each first image, the first image corresponds to the first text information one by one, all the first images are spliced ​​to obtain the first multi-entity sample data, all the first text information are spliced ​​to obtain the second multi-entity sample data, and the first multi-entity sample data and the second multi-entity sample data are encoded to obtain the multi-entity sample data. By splicing the single entity data to obtain multi-entity data, the training set is expanded so that the model can be better trained. Considering that the spliced ​​multi-entity data may have incoherent context and no connecting words, the recognition model is trained through a mask mechanism so that the model can understand the spliced ​​multi-entity data set and extract entities more accurately.

[0006] Furthermore, the specific step of obtaining the second image based on the comprehensive vector and the generator includes: combining the comprehensive vector with a random noise vector to obtain an input condition of the generator, and the generator generates the second image based on the input condition. The generator can be guided to generate an image that reflects the semantic information of the entity tag and has diversity.

[0007] Furthermore, the method further includes obtaining a discriminator to determine whether the discriminator can determine whether the second image is true or false. The specific steps include:

[0008] A1. Obtain the second loss L of the discriminator D ;

[0009] A2, based on the second loss L D Determining whether the discriminator can effectively distinguish the authenticity of the generated image;

[0010] A3. If not, based on the second loss L D Adjust the discriminator to obtain the optimized discriminator, return to A1, and update the discriminator to the optimized discriminator; if so, determine whether the generated image is true based on the discriminator.

[0011] A batch of real data and false data is given to the discriminator, and the loss is calculated based on the probability that it is judged to be true. Then, by adjusting the model parameters, the discriminator is given the function of judging whether the image is true or false, which can ensure that the generated image is a real image to a certain extent.

[0012] Further, based on the second image, a first loss L of the generator is obtained. G , based on the first loss L G The specific steps of obtaining the target generation parameters of the generator include:

[0013] B1, judging whether the second image is real based on the discriminator;

[0014] B2. If it is false, obtain the third loss L of the discriminator G1 , based on the third loss L G1 Get the first loss L G , based on the first loss L G Adjust the first generation parameter of the generator to obtain the second generation parameter, update the first generation parameter of the generator to the second generation parameter, the generator obtains the third image based on the second generation parameter, returns B1, and updates the second image to the third image; if true, obtain the fourth loss L of the second image G2 , based on the fourth loss L G2 Get the first loss L G , based on the first loss L G The first generation parameter is adjusted to obtain the target generation parameter.

[0015] Calculate the loss of the generator, the loss from the discriminator in the early stage, and the loss from the inconsistency between the number of entities in the generated image and the entity type in the later stage. First, adjust the image in the direction of minimum loss according to the loss of the discriminator, minimize information loss, and improve the quality of the generated image. Then, adjust the image in the direction of minimum loss according to the loss of the inconsistency between the number of entities and the entity type, minimize information loss, and improve the accuracy, diversity, and authenticity of the generated image.

[0016] Further, a fourth loss L of the second image is obtained G2 The specific steps include: obtaining the third text information of the second image; recognizing the second image and the third text information based on the recognition model to obtain a plurality of third entities and a plurality of fourth entities respectively; obtaining the number of all the third entities and all the fourth entities to obtain a first number and a second number respectively; obtaining a fifth loss L based on all the third entities and all the fourth entities e , a sixth loss L is obtained based on the first quantity and the second quantity mis , based on the fifth loss L e and the sixth loss L mis Obtain the fourth loss L G2 .

[0017] In the traditional text-to-image model, there is only the third loss L returned by the discriminator. G1 , focusing only on the quality of image generation, the generative model of this method combines the fourth loss L G2 , by the fifth loss L e and the sixth loss L mis Calculated, combined with the fourth loss L G2 The model will focus on the content of the generated image and ensure the semantic alignment of text and image, which not only generates high-quality images but also improves the accuracy, diversity and authenticity of the generated images.

[0018] Further, the second loss L is obtained D The first calculation method is:

[0019] L D =E x～data [max(0,1-D(x))]+E z～p(z) [max(0,1+D(z))];

[0020] Where D represents the discriminator, max(0,1-D(x)) represents whether the discriminator can effectively distinguish true images when judging real images, max(0,1+D(Z) represents whether the discriminator can effectively distinguish false images when judging generated images, x represents samples sampled from the real data distribution, data represents real data, z represents samples obtained from a specific distribution, p(z) represents a specific distribution, and E represents mathematical expectation.

[0021] Further, the third loss L is obtained G1 The second calculation method is:

[0022] L G1 =-E z～p(z) [D(G(z))];

[0023] Among them, D(G(z)) represents the probability value of whether the discriminator judges whether the generated image is real, G(z) represents the false sample generated by the generator, and G represents the generator.

[0024] Further, the fifth loss L is obtained e The third calculation method is:

[0025]

[0026] Get the sixth loss L mis The fourth calculation method is:

[0027] L mis =max(0,|NM| 2 );

[0028] Where N represents the first quantity, n ti represents the embedding vector corresponding to the i-th first entity, n ii represents the embedding vector corresponding to the i-th second entity, and M represents the second quantity.

[0029] Further, the fourth loss L is obtained G2 The fifth calculation method is:

[0030] L G2 =L mis +L e ;

[0031] One or more technical solutions provided by the present invention have at least the following technical effects or advantages:

[0032] 1. All the first images are spliced ​​to obtain first multi-entity sample data, all the first text information is spliced ​​to obtain second multi-entity sample data, the first multi-entity sample data and the second multi-entity sample data are encoded to obtain the multi-entity sample data, and a recognition model is obtained based on the multi-entity sample data. By splicing single entity data to obtain multi-entity data, the training set is expanded so that the model data set can be better trained, and the recognition model is trained through a mask mechanism so that the model can understand the spliced ​​multi-entity data set and extract entities more accurately.

[0033] 2. Based on the recognition model, the text data is identified to obtain several first entities, each first entity is encoded to obtain several single vectors, all the single vectors are merged to obtain a comprehensive vector, and the unstructured data of the text data is converted into structured data.

[0034] 3. Combine the comprehensive vector with the random noise vector to obtain the input condition of the generator. The generator generates a second image based on the input condition. The comprehensive vector captures the joint semantic information of all entity labels. The random noise vector is combined to guide the generator to produce images that reflect the semantic information of the entity labels and have diversity.

[0035] 4. Based on the second loss L D Determine whether the discriminator can effectively distinguish the authenticity of the generated image, calculate the loss based on the probability of its judgment being true, and then adjust the model parameters so that the discriminator has the function of judging the authenticity of the image, which can ensure that the generated image is a real image to a certain extent.

[0036] 5. The third loss L based on the discriminator G1 Get the first loss L of the generator G , based on the first loss L G Adjusting the generation parameters of the generator so that the image generated by the optimized generation parameters can be judged as a real image by the discriminator can reduce the information loss on unstructured structured data such as text and images. By continuously optimizing the generation parameters of the generator through calculating the loss, the generated image can be continuously adjusted and optimized to improve the image quality.

[0037] 6. Based on the fourth loss LG2, the first loss LG is obtained. Based on the first loss LG, the generation parameters of the generator are adjusted to obtain the target generation parameters. By calculating the loss of entity types and quantities of texts and generated images, the generation parameters of the generator are continuously optimized again through this loss, so as to continuously adjust and optimize the generated images to ensure the semantic alignment of text and image, which not only generates high-quality images, but also improves the accuracy, diversity and authenticity of the generated images. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The drawings described herein are used to provide a further understanding of the embodiments of the present invention, constitute a part of the present invention, and do not constitute a limitation on the embodiments of the present invention;

[0039] Figure 1 It is a flowchart of a method for generating an image from text in the present invention;

[0040] Figure 2 It is a specific flow chart of obtaining a generation model in a method for generating images from text in the present invention. DETAILED DESCRIPTION

[0041] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present invention and the features in the embodiments can be combined with each other without conflict.

[0042] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those within the scope of this description. Therefore, the protection scope of the present invention is not limited to the specific embodiments disclosed below.

[0043] Embodiment 1

[0044] refer to Figure 1 and Figure 2 , this embodiment provides a method for generating an image from text, the method comprising:

[0045] Acquire single entity sample data, obtain multi-entity sample data based on the single entity sample data, and obtain a recognition model based on the multi-entity sample data;

[0046] Among them, the specific steps of obtaining multi-entity sample data based on the single-entity sample data include: the single-entity sample data includes several first images and first text information of each of the first images, the first images correspond to the first text information one-to-one, all the first images are spliced ​​to obtain first multi-entity sample data, all the first text information are spliced ​​to obtain second multi-entity sample data, and the first multi-entity sample data and the second multi-entity sample data are encoded to obtain the multi-entity sample data.

[0047] For example, if there are two images, one of which is a brown wolf and the other is a black bear with brown nose hair, the two images are spliced ​​together, and the corresponding texts are also spliced ​​together. The text information is: "A brown wolf. A black bear with brown nose hair". Considering that there are no conjunctions and the context is incoherent, the mask mechanism is used to retrain the Bert model to obtain a recognition model so that the recognition model can understand the spliced ​​text;

[0048] Based on the recognition model, the multi-entity sample data is recognized to obtain a plurality of first entities, each of the first entities is encoded to obtain a plurality of single vectors, and all the single vectors are fused to obtain a comprehensive vector; based on the comprehensive vector and the generator, a second image is obtained, and based on the second image, a first loss L of the generator is obtained. G , based on the first loss L G Obtain target generation parameters of the generator, and obtain a generation model based on the target generation parameters; obtain text data, recognize the text data based on the recognition model to obtain a number of second entities, and the generation model obtains images corresponding to the text data based on all the second entities.

[0049] Among them, the specific steps of obtaining the second image based on the comprehensive vector and the generator include: weightedly combining the comprehensive vector with the random noise vector based on the conditional batch normalization layer of the GAN model to obtain the input condition of the generator, and the generator generates the second image based on the input condition.

[0050] Obtain a first loss L of the generator based on the second image G , based on the first loss L G The specific steps of obtaining the target generation parameters of the generator include:

[0051] B1, judging whether the second image is real based on the discriminator;

[0052] B2. If it is false, obtain the third loss L of the discriminator G1 , based on the third loss L G1 Get the first loss L G , based on the first loss L G Use the gradient descent algorithm to adjust the first generation parameter of the generator to obtain the second generation parameter, update the first generation parameter of the generator to the second generation parameter, the generator obtains the third image based on the second generation parameter, returns B1, and updates the second image to the third image; if true, obtain the fourth loss L of the second image G2 , based on the fourth loss L G2 Get the first loss L G , based on the first loss L G The first generation parameter is adjusted by using a gradient descent algorithm to obtain the target generation parameter.

[0053] Get the fourth loss L of the second image G2The specific steps include: obtaining the third text information of the second image; recognizing the second image and the third text information based on the recognition model to obtain a plurality of third entities and a plurality of fourth entities respectively;

[0054] Obtaining the quantities of all the third entities and all the fourth entities to obtain a first quantity and a second quantity respectively;

[0055] A fifth loss L is obtained based on all the third entities and all the fourth entities e , a sixth loss L is obtained based on the first quantity and the second quantity mis , based on the fifth loss L e and the sixth loss L mis Obtain the fourth loss L G2 .

[0056] The recognition model may further include an upsampling layer, based on which the second image is recognized to obtain all the second entities, and the second image may be sampled to a higher resolution without changing the deep feature information of the second image, retaining the original information, and providing a stable foundation for the deep feature learning of the recognition model.

[0057] Embodiment 2

[0058] refer to Figure 2 Based on the first embodiment, in this embodiment, the method further includes obtaining a discriminator to determine whether the discriminator can determine whether the second image is true or false. The specific steps include:

[0059] A1. Obtain the second loss L of the discriminator D ;

[0060] A2, based on the second loss L D Determining whether the discriminator can effectively distinguish the authenticity of the generated image;

[0061] A3. If not, based on the second loss L D Adjust the discriminator to obtain the optimized discriminator, return to A1, and update the discriminator to the optimized discriminator; if so, determine whether the generated image is true based on the discriminator.

[0062] The discriminator can be a binary classification model, which is used to determine whether the image is real data or false data. Then a batch of real data and false data will be input into the discriminator, and then the loss will be calculated based on the probability that the model judges it to be true. If the judgment is wrong, the model parameters are adjusted through the gradient descent algorithm to enable the model to have binary classification capabilities.

[0063] Embodiment 3

[0064] Based on the above embodiment, in this embodiment, the second loss L is obtained. D The first calculation method is:

[0065] L D =E x～data [max(0,1-D(x))]+E z～p(z) [max(0,1+D(z))];

[0066] Where D represents the discriminator, max(0,1-D(x)) represents whether the discriminator can effectively distinguish true images when judging real images, max(0,1+D(Z) represents whether the discriminator can effectively distinguish false images when judging generated images, x represents samples sampled from the real data distribution, data represents real data, z represents samples obtained from a specific distribution, p(z) represents a specific distribution, and E represents mathematical expectation.

[0067] Get the third loss L G1 The second calculation method is:

[0068] L G1 =-E z～p(z) [D(G(z))];

[0069] Among them, D(G(z)) represents the probability value of whether the discriminator judges whether the generated image is real, G(z) represents the false sample generated by the generator, and G represents the generator.

[0070] Get the fifth loss L e The third calculation method is:

[0071]

[0072] Get the sixth loss L mis The fourth calculation method is:

[0073] L ims =max(0,|NM| 2 );

[0074] Where N represents the first quantity, n ti represents the embedding vector corresponding to the i-th first entity, n ii represents the embedding vector corresponding to the i-th second entity, and M represents the second quantity.

[0075] Get the fourth loss L G2 The fifth calculation method is:

[0076] L G2 =L mis +L e .

[0077] Embodiment 4

[0078] On the basis of the above embodiment, in this embodiment, the specific process of obtaining the generation model may be:

[0079] Step 1: Use the discriminator to determine whether the generated image is a real image, that is, the image can make the discriminator think it is a real image to a certain extent. When it is judged to be a false image, the loss of the discriminator is obtained and fed back to the generator. The gradient descent algorithm is used to continuously optimize the generation parameters of the generator until the generated image can be judged as a real image;

[0080] Step 2: When the image generated by the generation parameters can be recognized as a real image by the discriminator, the entity type and number of the corresponding text are introduced, and its loss is calculated. The loss is also passed back to the generator, and the generation parameters of the generator are continuously optimized again using the gradient descent algorithm until the entities of the generated image correspond to the entities of the text, the final generation parameters are determined, and then the generation model is obtained based on the generation parameters.

[0081] First ensuring the quality of generated images and then ensuring the semantic alignment of text and image can not only further generate high-quality images, but also improve the accuracy, diversity and authenticity of generated images.

[0082] In this embodiment, the first and second steps can also be performed at the same time, and the losses of the discriminator, entity type, and entity number are obtained at the same time, and they are fed back to the generator, and then the generation parameters are optimized by the gradient descent algorithm. Then, the first loss L of the generator is G The calculation method can be:

[0083] L G =L G1 +L G2 .

[0084] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0085] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A method for generating an image from text, characterized in that: The method comprises: Acquire single entity sample data, obtain multi-entity sample data based on the single entity sample data, and obtain a recognition model based on the multi-entity sample data; Based on the recognition model, the multi-entity sample data is recognized to obtain a number of first entities, each of the first entities is encoded to obtain a number of single vectors, and all the single vectors are fused to obtain a comprehensive vector; based on the comprehensive vector and the generator, a second image is obtained, and based on the second image, a first loss L of the generator is obtained. G , based on the first loss L G Obtain target generation parameters of the generator, and obtain a generation model based on the target generation parameters; obtain text data, recognize the text data based on the recognition model to obtain a plurality of second entities, and obtain an image corresponding to the text data based on all the second entities using the generation model; The method further includes obtaining a discriminator and determining whether the discriminator can determine whether the second image is true or false. The specific steps include: A1. Obtain the second loss L of the discriminator D ; A2, based on the second loss L D Determining whether the discriminator can effectively distinguish the authenticity of the generated image; A3. If not, based on the second loss L D Adjust the discriminator to obtain an optimized discriminator, return to A1, and update the discriminator to the optimized discriminator; if so, determine whether the generated image is true based on the discriminator; Obtain a first loss L of the generator based on the second image G , based on the first loss L G The specific steps of obtaining the target generation parameters of the generator include: B1, judging whether the second image is real based on the discriminator; B2. If it is false, obtain the third loss L of the discriminator G1 , based on the third loss L G1 Get the first loss L G , based on the first loss L G Adjust the first generation parameter of the generator to obtain the second generation parameter, update the first generation parameter of the generator to the second generation parameter, the generator obtains the third image based on the second generation parameter, returns B1, and updates the second image to the third image; if true, obtain the fourth loss L of the second image G2 , based on the fourth loss L G2 Get the first loss L G , based on the first loss L G The first generation parameter is adjusted to obtain the target generation parameter.

2. The method for generating an image from text according to claim 1, characterized in that: The specific steps of obtaining multi-entity sample data based on the single-entity sample data include: The single entity sample data includes several first images and first text information of each of the first images, the first images correspond to the first text information one by one, all the first images are spliced ​​together to obtain first multi-entity sample data, all the first text information are spliced ​​together to obtain second multi-entity sample data, and the first multi-entity sample data and the second multi-entity sample data are encoded to obtain the multi-entity sample data.

3. The method for generating an image from text according to claim 1, characterized in that: The specific steps of obtaining the second image based on the comprehensive vector and the generator include: The comprehensive vector is combined with a random noise vector to obtain an input condition of the generator, and the generator generates the second image based on the input condition.

4. The method for generating an image from text according to claim 1, characterized in that: Get the fourth loss L of the second image G2 The specific steps include: Acquire third text information of the second image; recognize the second image and the third text information based on the recognition model to obtain a plurality of third entities and a plurality of fourth entities respectively; Obtaining the quantities of all the third entities and all the fourth entities to obtain a first quantity and a second quantity respectively; A fifth loss L is obtained based on all the third entities and all the fourth entities e , a sixth loss L is obtained based on the first quantity and the second quantity mis , based on the fifth loss L e and the sixth loss L mis Obtain the fourth loss L G2 .

5. The method for generating an image from text according to claim 4, characterized in that: Get the second loss L D The first calculation method is: L D =E x～data [max(0,1-D(x))]+E z～p(z) [max(0,1+D(z))]; Where D represents the discriminator, max(0,1-D(x)) represents whether the discriminator can effectively distinguish true images from true images, max(0,1+D(Z)) represents whether the discriminator can effectively distinguish false images from false images, x represents samples sampled from the real data distribution, data represents real data, z represents samples sampled from a specific distribution, p(z) represents a specific distribution, and E represents mathematical expectation.

6. The method for generating an image from text according to claim 5, characterized in that: Get the third loss L G1 The second calculation method is: L G1 =-E z～p(z) [D(G(z))]; Among them, D(G(z)) represents the probability value of whether the discriminator judges whether the generated image is real, G(z) represents the false sample generated by the generator, and G represents the generator.

7. The method for generating an image from text according to claim 6, characterized in that: Get the sixth loss L mis The third calculation method is: L mis =max(0,|NM| 2 ); Among them, N represents the first number, and M represents the second number.

Images