A method and device for generating images from texts by introducing class information

Abstract

The present invention discloses a method and device for generating images by introducing class information from text, wherein the method comprises a training phase and a testing phase. The training phase is based on a generative adversarial network, and uses natural language text describing an image, the class label of the text, and the corresponding real image to train a generator and a discriminator; the testing phase uses the text and its class label to generate a corresponding image in the generator. The beneficial effects of the present invention are as follows: according to text information encoding and class information encoding, text semantic image features and class information image features are generated by transcoding respectively, and then the two levels of image features are fused to decode and generate an image. In the process of generating an image, the corresponding class information is introduced to strengthen the correlation between the generated image and the text. At the same time, in the training process, a multi-stage generation process is used to gradually generate higher resolution images, thereby reducing the training difficulty of directly generating high-resolution images.

Description

Technical Field

[0001] The present invention relates to the technical field of deep learning generative models, and in particular to a method and device for generating images from text by introducing class information. Background Art

[0002] Generating images from text is an important problem and has a wide range of applications, such as computer-aided medical treatment, news picture generation, etc.

[0003] The research on text-to-image generation methods is mainly based on two generative models, namely Conditional Variational Auto-Encoder (CVAE) and Conditional Generative Adversarial Networks (CGAN). The images generated by the CVAE method often have the problem of image blur, and the current mainstream methods are all based on the CGAN model.

[0004] Due to the instability of GAN training itself, it is very difficult to directly generate high-resolution images from text descriptions. Therefore, the stacked generative adversarial network (StackGAN) proposed a strategy of first generating low-resolution images from text, and then gradually generating high-resolution images from low-resolution images, which was widely used in subsequent work.

[0005] The problem of generating images from text can be divided into two main sub-problems: 1) How to capture the semantic representation of text, generally by extracting text semantic embedding and visual related information in the text description through a text encoder; 2) How to use the text semantic representation in 1 to generate a realistic image that matches the text through a generator, so that humans mistakenly believe it is a real and related picture.

[0006] Traditional text-generated image networks only use text information and ignore the class information of the text itself. However, this class information is also helpful for text generation. Objects of the same class often have certain similarities. Introducing the class information of the text can help solve the problem of the one-sidedness of a single text description. In addition, it can narrow the correlation between the generated image and the text. For methods of generating images from text, the Inception Score (IS) indicator is widely used for evaluation. IS evaluates the quality of the generated image by calculating the correlation between the distribution of the generated image and the distribution of the real image. The higher the IS value, the clearer and more recognizable entities the generated image contains. Summary of the invention

[0007] In view of the shortcomings of the above-mentioned existing image generation methods, the present invention provides a method and device for generating images from text by introducing class information, which can introduce the class information to which the text belongs during the image generation process, constrain the relevance of images generated by the same class of text through the class information, and solve the problem that a single text description may be incomplete.

[0008] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0009] A method and device for generating images from texts by introducing class information, characterized in that the method comprises the following steps:

[0010] Step 1: Encode the natural language text describing the image to obtain the text semantic embedding representation;

[0011] Step 2: Encode the class label of the text itself to obtain the semantic embedding representation of the class information;

[0012] Step 3: Mix the text semantic embedding representation obtained in step 1 with random noise, use a recurrent neural network to read the text semantic embedding representation and random noise, and output the object latent code of the text;

[0013] Step 4: Mix the semantic embedding representation of the class information obtained in step 2 with the noise, and obtain the object latent code of the class information through variational inference;

[0014] Step 5, decoding the text latent code and class information latent code obtained in step 3 and step 4 respectively to obtain text-level image features and class-level image features;

[0015] Step 6, decoding the fused text-level image features and class-level image features obtained in step 5 to generate an image;

[0016] Step 7, performing adversarial training on the generated image obtained in step 6 and the corresponding real image;

[0017] Step 8: upsample the image features obtained in step 5 to obtain image features of different dimensions, and repeat steps 6-7 to gradually generate images with higher resolutions;

[0018] Step 9: In the testing phase, input the text and its class label, repeat steps 1-6, and generate high-resolution images through multiple stages in the image generator.

[0019] Furthermore, in step 1, the method for encoding the natural language text describing the image is: segmenting the natural language text to obtain a word sequence p with a length of d = (w1, w2, ... w d ), where each word w iUse pre-trained word vectors for representation, i = 1 ~ d, and use the obtained word vectors to encode the text;

[0020] Furthermore, in step 2, if each text-image data belongs to only one class, the class information is encoded using a one-hot encoding method; if the text-image data terms belong to multiple classes, the class information is encoded using a multi-hot encoding method.

[0021] Furthermore, in step 3, the recurrent neural network adopts a long short-term memory network.

[0022] Furthermore, in step 3, the mixing method of text semantic embedding representation and noise adopts a direct connection method, the noise adopted is Gaussian noise z~N(0,I), and the mixing result of text semantic embedding s and z is z s =(s,z).

[0023] Furthermore, the hybrid method of class information semantic embedding and noise in step 4 is variational inference, that is, the variational encoder infers the latent attribute distribution q(z c |c,z), the class-informed semantic embedding representation z sampled from this distribution c .

[0024] Furthermore, in step 5, an upsampling operation is used to decode the text latent code and the class information latent code to obtain image features.

[0025] Furthermore, in step 6, the image feature h generated by the text c The image features h generated by the class information r The fusion is performed by cascading, and the fused image features can be expressed as h1=h c ⊙h r ; Use convolutional neural network to decode the fused image features to generate an image.

[0026] Furthermore, the adversarial training method in step 7 is: the generated image and the real image are respectively obtained through a convolutional neural network to obtain the image latent representation, and the corresponding text and class information are input at the same time, and the scores of the image authenticity, the degree of matching between the image and the text, and the degree of matching between the image and the class information are output.

[0027] Furthermore, in step 8, a phased image generation method is used to gradually generate a higher resolution image. Taking the two-stage image generation as an example, the first stage uses the fused image features to generate a low-resolution image; the second stage generates the image features h generated by the text obtained in the first stage. c The image features h generated by the class informationr Upsampling is performed to obtain higher-dimensional image features and text features, and then generate higher-resolution images.

[0028] Furthermore, in the two-stage image generation network, the resolution of the image generated in the first stage is 64*64, and the resolution of the image generated in the second stage is 128*128, and the model can continue to be stacked.

[0029] Furthermore, the input of the test phase in step 9 is text and its class label, and the generator model obtained in the training phase generates high-resolution images in stages.

[0030] A method and device for generating an image from text by introducing class information, characterized in that the device comprises:

[0031] Text encoder, which encodes the text describing the image to obtain the text semantic embedding representation;

[0032] The class information encoder encodes the class information of the text describing the image to obtain the semantic embedding representation of the class information;

[0033] The generator includes a recurrent neural network transcoder and a variational inference transcoder, an image feature fusion device and an image decoder. The recurrent neural network transcoder is used to read the text semantic embedding and the hidden state of the previous step of the transcoder, and output the corresponding text image features; the variational inference transcoder is used to read the input class information semantic embedding and output the corresponding class information image features; the image feature fusion device fuses the text image features and class information image features generated by the recurrent neural network transcoder and the variational inference; the image decoder decodes the input fused image features to generate an image;

[0034] A discriminator, comprising an image semantic discriminator, a text semantic discriminator and a class information discriminator, wherein the image semantic discriminator determines the correlation between the generated image and the real image; the text semantic discriminator determines the correlation between the generated image and the corresponding text; and the class information discriminator determines the correlation between the generated image and the class information. A discriminator, comprising an image semantic discriminator, a text semantic discriminator and a class information discriminator, wherein the image semantic discriminator determines the correlation between the generated image and the real image; the text semantic discriminator determines the correlation between the generated image and the corresponding text; and the class information discriminator determines the correlation between the generated image and the class information.

[0035] Compared with the prior art, the present invention has the following beneficial effects:

[0036] The present invention introduces additional class information in the process of generating images from text, such as the class label of the bird itself and the labels of different objects contained in the picture. The class information obtains its implicit code through the variational inference method, which can fully mine the entity information behind the class label. The image features generated by the text and the image features generated by the class information are fused in the image space, reducing the difficulty of training. This method can enrich the semantics of the generated image by capturing additional class information, and at the same time generate pictures that are more relevant to the text. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 It is a flow chart of the generator method of the present invention.

[0038] Figure 2 It is a flow chart of the method of the discriminator in the present invention. DETAILED DESCRIPTION

[0039] The present invention is further explained below in conjunction with the accompanying drawings and specific embodiments. It should be understood that these examples are only used to illustrate the present invention and are not used to limit the scope of the present invention. After reading the present invention, various equivalent forms of modifications to the present invention by those skilled in the art fall within the scope defined by the claims attached to this application. A method for generating images from text based on conditional generation of adversarial text by introducing class information, such as Figure 1-2 As shown, the following steps are included:

[0040] Step 1: Construct a text encoding unit, including a text encoder and a recurrent neural network transcoder, such as Figure 1 As described in S1. The text encoder inputs natural language text and outputs the embedded representation of the text. The natural language text here uses English text, and after removing stop words, a word sequence of length d is obtained, in which each word is represented by a pre-trained word vector.

[0041] For example, when inputting the natural language text “This little bird is almost blue with black primary and secondaries”, after removing stop words, the final word sequence is [little, bird, almost, blue, black, primary, secondaryaries], d = 7, we set the maximum value of d to 18, fill in the missing part and truncate the excess part.

[0042] The goal of the recurrent neural network encoder is to extract high-level semantic features from natural language texts, and it is performed by a pre-trained bidirectional long short-term memory network (Bi-LSTM). The input text sequence is the hidden state h of each word output. i As the word-level feature of the word, the average of the hidden time sequence output at all moments is used as the semantic embedding of the text, that is,

[0043] What is given here is only a preferred method of text encoding unit, and other reasonable encoding methods can also be used for encoding.

[0044] Step 2: construct a class information encoding unit, including a class information encoder and a variational inference encoder, such as Figure 1 As described in S2. The class information encoder inputs the class information corresponding to the text and outputs the embedded representation of the class information. The class information of the text here has two cases: single class and multi-class. If each text itself has only one class label, we use one-hot to encode the class information, such as "This little bird is almost blue with black primary and secondaries." There is only one class label Indigo, which represents the type of bird corresponding to the text. There are 20 different types of birds in the data set, so we encode the class information into a 20-dimensional one-hot vector [1,0,…0,0]; if the text itself has multiple class attributes, it is encoded using multi-hot encoding, such as [1,0,0,1,0], which means that the text has the first and third class labels.

[0045] After encoding the class information into the class vector c, we use variational inference to convert the class vector into class information embedding. As a condition, given z and We assume that the posterior distribution of the latent variable is It obeys a multivariate diagonal Gaussian distribution, where the mean and variance of the latent distribution are learned by the encoder. We use a three-layer linear neural network for inference here, and we can also use more complex encoding methods according to the distribution of classes in the data.

[0046] Step 3: Construct a module to fuse text information and class information, such as Figure 1 As described in S3. This module fuses the image features generated by text and the image features generated by class information. The fusion module consists of an upsampling network that extracts image features and a concatenation of text image features and class information image features. For text information, the upsampling network inputs the joint data z of text semantic embedding s and noise s , upsample to get the image feature h of the corresponding dimension s ; For class information, the upsampling network infers the posterior distribution of the latent variables from the class information encoder The input z is sampled c , get the class information image feature h of the same dimension c; Finally, we perform dot multiplication on the text information image feature and the class information image feature to obtain the final fused image feature h.

[0047] Step 4: Construct a conditional generative adversarial network consisting of a generator and a discriminator, such as Figure 1 S4 and Figure 2 The generator is composed of a convolutional neural network, and the discriminator is composed of an image discriminator, a text semantic discriminator, and a class information discriminator. The generator uses a scale-invariant convolutional layer to decode the image and transform the fused image features into the final generated image; the image discriminator in the discriminator scores the authenticity of the generated image, the text semantic discriminator is used to evaluate the correlation between the generated image and the original text, and the class information discriminator scores the matching degree of the class information of the generated image.

[0048] Step 5, inputting the natural language text describing the image and the corresponding class information into the text encoder and the class information encoder respectively to obtain the text semantic embedding representation and the class information embedding representation;

[0049] Step 6, output the generated text semantic embedding representation and class information embedding representation to the text information and class information fusion module to obtain the image features that fuse the two types of information;

[0050] Step 7: Input the fused image features into the image generator to generate a lower resolution image. Here, the resolution is set to 64*64. Input the corresponding real image, natural language text, and class information into the discriminator for adversarial training. The loss functions of the generator and discriminator during the adversarial training are as follows:

[0051] L D =-(E x～P [log[D(x) r ]+E x～Q [1-log[D(x) r ])

[0052] -(E x～P [log[D(x) c ]+E x～Q [1-log[D(x) c ])

[0053] -(E x～P [log[D(x,s)]+E x～Q [1-log[D(x,s)])

[0054]

[0055] In the formula, P is the real data distribution, Q is the generated data distribution, and D(x) rrepresents the probability that the generated image x is real, D(x) c represents the probability that the generated image belongs to the correct class label, and D(x,s) represents the matching probability between the generated image and the description text.

[0056] Two KL divergence terms are added to the generator’s loss function as constraints on the two latent variables z c and z s During the training process, first, with the generator fixed, the loss L D Optimize the discriminator D(x), and then fix the discriminator according to the loss L G Optimize the generator G. The above two steps are trained alternately through mini-batch stochastic gradient descent.

[0057] Step 8: Upsample the image features, text image features and class information image features generated in the first stage to 128*128 dimensions respectively. Repeat the adversarial training process in step 7 in a higher dimension to generate images with higher resolution.

[0058] When training this network, normalization techniques such as batch normalization and spectral normalization can be added to the generator and discriminator to stabilize the training and further improve the generation quality.

[0059] In summary, the text generation method of introducing class information disclosed in the present invention adds a module for encoding class information and fusing class information with text information compared to previous methods. The present invention introduces the category mark of the text itself, restricts the category of the generated image in the discriminator, and at the same time improves the correlation between the generated image and the text by introducing the class information.

[0060] In the experimental process, the benchmark model of the present invention is based on the StackGAN. The dimensions of the latent variables and the noise variables are set to 128. The discriminator is trained once and the generator is trained once for each iteration of the adversarial training. The network is trained using the Adam solver, where β1=0.5, β2=0.999; and the learning rate α=0.0002.

[0061] On the CUB dataset, IS is improved from 3.35±0.02 to 3.74±0.03; on the COCO dataset, IS is improved from 7.34±0.17 to 7.46±0.30. The image generation quality and entity clarity in the generated images are better than the baseline model.

[0062] The above embodiments are only preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited by the above embodiments. It should be pointed out that for those skilled in the art, without departing from the principle of the present invention, several improvements, modifications, substitutions, combinations and simplifications can be made, which are all equivalent replacement methods and should also be regarded as the protection scope of the present invention.

Claims

1. A method for generating images from text by introducing class information, characterized in that: The method comprises the following steps: Step 1, encode the natural language text describing the image to obtain a text semantic embedding representation; in step 1, the method for encoding the natural language text describing the image is: segment the natural language text to obtain a word sequence p with a length of d = (w1, w2, ... w d ), where each word w i Use pre-trained word vectors for representation, i = 1 ~ d, and use the obtained word vectors to encode the text; Step 2, encoding the class label of the text itself to obtain a semantic embedding representation of the class information; in step 2, if each text-image data belongs to only one class, the class information is encoded using a one-bit effective encoding method; if the text-image data belongs to multiple classes, the class information is encoded using a multi-bit effective encoding method; Step 3, the text semantic embedding representation obtained in step 1 is mixed with random noise, and a recurrent neural network is used to read the text semantic embedding representation and random noise, and the object latent code of the text is output; in step 3, the mixing method of the text semantic embedding representation and the noise is a direct connection method, the noise used is Gaussian noise z~N(0,I), and the mixing result of the text semantic embedding s and z is z s =(s,z); Step 4, the semantic embedding representation of the class information obtained in step 2 is mixed with the noise, and the object latent encoding of the class information is obtained by variational inference; the mixing method of the semantic embedding of the class information and the noise in step 4 is variational inference, that is, the variational encoder infers the latent attribute distribution q(z) of the class information when the noise z and the class information c are determined. c |c,z),z c represents the class-informed semantic embedding sampled from this distribution; Step 5, decoding the text latent code and class information latent code obtained in step 3 and step 4 respectively to obtain text-level image features and class-level image features; Step 6, decoding the fused text-level image features and class-level image features obtained in step 5 to generate an image; Step 7, performing adversarial training on the generated image obtained in step 6 and the corresponding real image; Step 8: upsample the image features obtained in step 5 to obtain image features of different dimensions, and repeat steps 6-7 to gradually generate images with higher resolutions; Step 9: In the testing phase, input the text and its class label, repeat steps 1-6, and generate high-resolution images through multiple stages in the image generator.

2. The method for generating images from texts with imported class information according to claim 1, characterized in that: In step 3, the recurrent neural network adopts a bidirectional long short-term memory network.

3. The method for generating images from texts with imported class information according to claim 1, characterized in that: In step 5, an upsampling operation is used to decode the text latent code and the class information latent code to obtain image features.

4. The method for generating images from texts with imported class information according to claim 1, characterized in that: In step 6, the image feature h generated by the text c The image features h generated by the class information r The fusion is performed by dot multiplication. The fused image features can be expressed as h1=h c ⊙h r ; Use convolutional neural network to decode the fused image features to generate an image.

5. The method for generating images from texts with imported class information according to claim 1, characterized in that: The adversarial training method in step 7 is: obtain image latent representations of the generated image and the real image through a convolutional neural network respectively, input the corresponding text and class information at the same time, and output scores for the authenticity of the image, the degree of matching between the image and the text, and the degree of matching between the image and the class information.

6. The method for generating images from texts with imported class information according to claim 1, characterized in that: In step 8, a staged image generation method is used to gradually generate a higher resolution image. Taking the two-stage image generation as an example, the first stage uses the fused image features to generate a low-resolution image; the second stage generates the image features h generated by the text obtained in the first stage. s The image features h generated by the class information c Upsampling is performed to obtain higher-dimensional image features and text features, and then generate higher-resolution images.

7. The method for generating images from texts with imported class information according to claim 6, characterized in that: In the two-stage image generation network, the resolution of the image generated in the first stage is 64*64, and the resolution of the image generated in the second stage is 128*128.

8. The method for generating images from texts with imported class information according to claim 1, characterized in that: The input of the test phase in step 9 is text and its class label, and the generator model obtained in the training phase generates high-resolution images in stages.

9. A text-generated image device that introduces class information, characterized in that: The device comprises: The text encoder encodes the text describing the image to obtain the text semantic embedding representation; the method for encoding the natural language text describing the image is: segment the natural language text to obtain a word sequence p = (w1, w2, ... w d ), where each word w i Use pre-trained word vectors for representation, i = 1 ~ d, and use the obtained word vectors to encode the text; The class information encoder encodes the class information of the text describing the image to obtain the semantic embedding representation of the class information; if each text-image data belongs to only one class, the class information is encoded using a one-bit effective encoding method; if the text-image data belongs to multiple classes, the class information is encoded using a multi-bit effective encoding method; The generator includes a recurrent neural network transcoder and a variational inference transcoder, an image feature fusion device and an image decoder. The recurrent neural network transcoder is used to read the text semantic embedding and the hidden state of the previous step of the transcoder, and output the corresponding text image features; the variational inference transcoder is used to read the input class information semantic embedding and output the corresponding class information image features; the image feature fusion device fuses the text image features and class information image features generated by the recurrent neural network transcoder and the variational inference; the image decoder decodes the input fused image features to generate an image; the mixing method of the text semantic embedding representation and the noise adopts a direct connection method, the noise adopted is Gaussian noise z~N(0,I), and the mixing result of the text semantic embedding s and z is z s =(s,z); The discriminator includes an image semantic discriminator, a text semantic discriminator and a class information discriminator. The image semantic discriminator determines the correlation between the generated image and the real image; the text semantic discriminator determines the correlation between the generated image and the corresponding text; the class information discriminator determines the correlation between the generated image and the class information; the hybrid method of class information semantic embedding and noise is variational inference, that is, the variational encoder infers the hidden attribute distribution q(z) of the class information when the noise z and the class information c are determined. c |c,z),z c represents the class-informative semantic embedding sampled from this distribution.

Images