Text image generation method based on twinborn-word level discriminator

Abstract

The invention discloses a text image generation method based on a twinborn-word level discriminator, particularly relates to the technical field of text image generation, and solves the problem that in the prior art, a common method of a condition discriminator only considers the influence of global sentence features in text description on image generation. The technical problems that the matching degree of fine-grained information of a certain word and the text is easily missed due to the fact that the sub-regions related to the word and the image in the text are not involved, and then the discrimination effect of a discriminator is affected are solved. According to the technical scheme, the method comprises the following steps of: aligning identification information, semantic information and real image information of a generated image in a training process; the quality effect of the generated image can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of text-to-image generation, and in particular to a text-to-image generation method based on a twin-word level discriminator. Background Art

[0002] The task of generating images from text not only requires computers to understand the semantic information in the text description, but also requires them to generate image information that matches the text's semantic information and conforms to human visual perception. This is a challenging task at the intersection of natural language and computer vision. In recent years, with the introduction and rapid development of generative adversarial networks, their powerful generation capabilities have provided new research ideas for the task of generating images from text.

[0003] In the task of generating images from text based on generative adversarial networks, the generator usually receives more attention, while the research on the discriminator is relatively neglected. The discriminator plays a vital role in generative adversarial networks (GANs). A conditional discriminator is usually used to distinguish whether the generated and real images and text descriptions match. Therefore, the ability of the discriminator is enhanced by two adversarial losses between the generated image and the real image and between the image and the text, so as to better guide the generator to generate images that match the text description. The common method of the conditional discriminator is to extract embedded features from the text and the image respectively, and then fuse the features of the two, and directly train the discriminator on this basis. However, this method only considers the influence of the global sentence features in the text description on the generated image, while the sub-regions related to the words in the text and the image are not involved, which is easy to miss the matching degree of the fine-grained information of a word with the text, thereby affecting the discriminant effect. Therefore, it is necessary to enhance the discriminant ability of the discriminator on the matching of the image sub-region and the text word, so as to associate the fine-grained features of image generation with the words. Summary of the invention

[0004] Therefore, the present invention solves the technical problem that the commonly used method of conditional discriminator in the prior art only considers the influence of global sentence features in text description on generated images, while the sub-regions related to words in the text and the image are not involved, which easily misses the matching degree between the fine-grained information of a certain word and the text, thereby affecting the discrimination effect of the discriminator; the present invention provides a text-based image generation method based on a twin-word-level discriminator, which improves the quality of the generated image by aligning the identification information, semantic information and real image information of the generated image during the training process.

[0005] The present invention provides a method for generating images from text based on a twin-word level discriminator, comprising the following steps:

[0006] S1: A multi-stage attention generative adversarial network is used as the basic network structure, including a text encoder, an image encoder, a generator, and a discriminator. The text encoder is a pre-trained bidirectional LSTM network that encodes the text description T, which consists of sentence features s and word features w. It generates images of different resolutions from coarse to fine in three stages.

[0007] S2: Enhance the sentence feature s through a conditional enhancement technique, and then use the enhanced sentence feature F ca (s) is connected with the random vector z as the input of the first-stage generator G0; the conditional enhancement technique uses the commonly used (CA) to enhance the text description;

[0008] S3: In the second and third stage generators, an attention model is used to encode the word feature vector w and the hidden visual features n of the previous level generated graph i-1 Form an attention vector Fiattn(w,ni-1) and use them as the input of the corresponding generator to synthesize images of the corresponding resolution;

[0009] S4: Add a twin discriminant module to the discriminator to learn the similarity between the generated image and the real image, so as to fully explore the content semantic information of the real image;

[0010] S5: Generate image I according to the text description s , and randomly selected images that do not match the text description I s′ , using the twin mechanism to transform the positive sample into real I s and I s The distance between the two is shortened, and the negative sample I s′ and I s The distance is increased; the contrast loss is used to train the proposed twin structure; the contrast loss at each stage is defined as:

[0011]

[0012] d(I s ,I s ) 2 Represents the distance between the generated image and the real image after the twin mechanism. δ is used to identify whether the image is a positive sample or a negative sample, that is, 0 represents a negative sample and 1 represents a positive sample. Hyperparameter It is used to balance the distance value when δ = 0, and is set to 1.0 in this experiment;

[0013] S6: Use word-level discriminators to model the fine-grained associations between words in the text and the corresponding sub-regions of the image, and establish semantic-visual correlations between words and image sub-regions; the input of the word-level discriminator includes two aspects: text features and visual features.

[0014] Furthermore, in step S6, for the text feature, the word feature w, and sentence vector s, where w and The word features are respectively from the given text description T and the randomly selected word features that do not match the given text description T and are encoded;

[0015] For visual features, the real image I obtained by encoding based on the image encoder s and generate images and I s ; Using p i,j Represents visual features I s and I s , w={w k |k=1,...,L} represents word level w and Features, calculate contextual features of different image regions and words in text descriptions:

[0016]

[0017] Where m k is the attention weight of the kth word, p i,j A feature vector representing a sub-region of the image;

[0018] All word context correlations are summed to calculate the final correlation value between image I and sentence T:

[0019]

[0020] The context feature L content , visual features p i,j And the global sentence feature s is connected as the input of the discriminator to determine whether the input image is true or false.

[0021] Furthermore, the generation loss and discrimination loss of the three stages in step S1 are defined as:

[0022]

[0023] i is the stage value, i is 0 for the first stage, 1 for the second stage, and 2 for the third stage;

[0024] The unconditional loss is used to train the generation of images that match the real data distribution to fool the discriminator, while the conditional loss is trained to generate sample images that match the text description;

[0025] Discriminator D i The training is done by minimizing the cross entropy loss to classify the input into real or fake images:

[0026]

[0027] I i is the distribution of real data samples at different stages I i is the distribution of generated data samples from the same stage

[0028] Furthermore, in order to generate semantically consistent and realistic images, the final objective function of the generator and discriminator is defined as:

[0029]

[0030] K represents the number of stages; α i , λ0 and λ2 are hyperparameters that control different losses; The twin mechanism is used to improve the consistency of image generation; content Improving the discriminative ability of the discriminator from the word level guides the generator to generate images that are consistent with the given text description; In the loss, set α1=1, α2=2 and α3=3. DAMSM It is a text-image matching score measured based on cosine similarity.

[0031] Furthermore, the image generation process of the generator in step S3 is as follows:

[0032] n0=F0(z,F ca (s));

[0033] n i =F i (n i-1 ,F i attn (w,n i-1 )),i=1,2;

[0034] I i =G i (n i ),i=0,1,2

[0035] Among them, {n0,n1,n2} represent the hidden layer features at different stages, and {I0,I1,I2} represent the generated images of different scales: 64×64, 128×128 and 256×256.

[0036] Furthermore, in step S5, an adjustment parameter is added to the contrast loss. and Adaptively adjust the weights of positive and negative sample pairs; the sliding loss is defined as:

[0037]

[0038] γ represents an adjustable sliding hyperparameter; when the sliding hyperparameter γ=0, the sliding loss represents the loss; when γ>0, the greater the distance between the positive sample pairs, the greater the penalty weight. Therefore, setting γ>0 can effectively reduce the relative loss of good pairings and pay more attention to the matching of difficult samples.

[0039] In the above technical solution, the technical effects and advantages provided by the present invention are:

[0040] 1. The present invention adds a twin discriminant module to the original discriminator model, maximizes the distance between the generated image and the image that does not match the given text description, and minimizes the distance between the real image and the generated image to perform network training. During the training process, the identification information and semantic information of the generated image are aligned with the real image information, thereby improving the quality of the generated image.

[0041] 2. The word-level discriminant module in the present invention performs association modeling on words and corresponding image sub-regions, and then the generator generates corresponding images based on the gradient information provided by the discriminator. The word-level discriminant module can well improve the semantic consistency between the generated image and the corresponding text description. The text-to-image method of the present invention improves the semantic consistency and quality of the generated image. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0043] Figure 1 A schematic diagram of the structure of the text-to-image generation method of the twin-word-level discriminator of the present invention;

[0044] Figure 2 This is a structural diagram of the twin discriminant module in the discriminator of the present invention;

[0045] Figure 3 The discriminator structure diagram of the word-level discriminant module of the present invention;

[0046] Figure 4 A comparison of sample images generated from a set of CUB datasets using the existing method and the method of the present invention;

[0047] Figure 5 A comparison chart of sample images generated from a set of COCO datasets using the existing method and the method of the present invention. DETAILED DESCRIPTION

[0048] In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings.

[0049] Embodiment 1:

[0050] See also Figure 1 , as shown in the structural diagram of a text-to-image method based on a twin-word-level discriminator, which includes the basic model attention generative adversarial network, twin discriminant module, word-level discriminant module, generation loss and discrimination loss.

[0051] A multi-stage attention generative adversarial network (AttnGAN) is used as the basic network structure of the present invention. AttnGAN uses a tree-type generative adversarial network, including a text encoder, an image encoder, a generator, and a discriminator. The text encoder is a pre-trained bidirectional LSTM network to encode the text description T, which consists of sentence features s and word features w, and generates images of different resolutions from coarse to fine through three stages.

[0052] First, a conditional enhancement technique is used to enhance the sentence feature s, and then the enhanced sentence feature F is used ca (s) is concatenated with the random vector z and used as the input of the first-stage generator G0.

[0053] Secondly, in the second and third stage generators, an attention model is used to encode the word feature vector w and the hidden visual features n of the previous level generated graph i-1 Form an attention vector Fiattn(w,ni-1), and use them as the input of the corresponding generator to synthesize images of corresponding resolution. The image generation process of the generator is as follows:

[0054] n0=F0(z,F ca (s));

[0055] n i =F i (n i-1 ,F i attn (w,n i-1 )),i=1,2;

[0056] I i =G i (n i ),i=0,1,2

[0057] Among them, {n0,n1,n2} represent the hidden layer features at different stages, and {I0,I1,I2} represent the generated images of different scales: 64×64, 128×128 and 256×256.

[0058] In order to improve the consistency of visual semantics, a twinning mechanism is added to the multi-stage discriminator to enhance the consistency of the generated images. The discriminator is trained by minimizing the distance between the generated image and the real image and maximizing the distance between the generated image and the fake real image through contrast loss. Figure 2 The twin discriminator module is shown in the discriminator structure diagram. Given a text description, it generates an image I s , and randomly selected images that do not match the text description I s′ , using the twin mechanism to transform the positive sample into real I s and I s The distance between the two is shortened, and the negative sample I s′ and I s The distance is increased. The proposed twin structure is trained using contrast loss. The contrast loss at each stage is defined as:

[0059]

[0060] d(I s ,I s ) 2 Represents the distance between the generated image and the real image after the twin mechanism. δ is used to identify whether the image is a positive sample or a negative sample, that is, 0 represents a negative sample and 1 represents a positive sample. Using hyperparameters To balance the distance value when δ = 0, it is set to 1.0 in this experiment.

[0061] Contrastive loss is used to strengthen the training process of the twin network. However, during the training process, some samples are easy to distinguish, while some are difficult to distinguish. The focal loss function can well balance the training of simple and complex samples, reduce the loss of easy-to-classify samples, and make the model pay more attention to those difficult and complex samples. Therefore, in order to better train the twin module, inspired by the focal loss, an adjustment parameter is added to the contrast loss. and Adaptively adjust the weights of positive and negative sample pairs. Define it as sliding loss:

[0062]

[0063] γ represents an adjustable sliding hyperparameter. When the sliding hyperparameter γ = 0, the sliding loss represents the loss. When γ > 0, the greater the distance between the positive sample pairs, the greater the penalty weight. Therefore, setting γ>0 can effectively reduce the relative loss of good pairings and focus more on the matching of difficult samples.

[0064] In order to enhance the semantic consistency between the generated image of the generator and the given text description, the discriminator should provide better fine-grained gradient information to the generator. The feedback of this gradient information can better guide the generator to generate images of sub-regions corresponding to the most relevant words in the text. Figure 3 As shown in the discriminator structure diagram, a word-level discriminant module is introduced in the discriminator to explore the correlation between the image sub-region and each word in the sentence to guide the generation process of the generator.

[0065] In the word-level discriminant module, the text encoder encodes the text description into a global sentence vector and a word vector respectively, and uses the pre-trained Inception-V3 network as an image encoder to extract the visual features of the generated image and the real image. Therefore, the input of the word-level discriminator includes two aspects: text features and visual features. For text features, the word features w, and sentence vector s, where w and w come from the given text description T and the randomly selected word features encoded after not matching the given text description T. For visual features, the real image I encoded by the image encoder s and generate images and Specifically, using p i,j Represents visual features I s and I s , w={w k |k=1,...,L} represents word level w and Features, calculate contextual features of different image regions and words in text descriptions:

[0066]

[0067] m k represents the attention weight of the kth word, p i,j A feature vector representing a subregion of an image.

[0068] The final correlation value L between image I and sentence T i,j is calculated by summing up all word contexts:

[0069]

[0070] The context feature L content , visual features p i,j And the global sentence feature s is connected as the input of the discriminator to judge whether the input image is real or fake. By doing so, the generator can receive fine-grained feedback on each visual attribute from the word-level discriminator, which can further help it supervise the generation and operation of each sub-region independently.

[0071] In the three stages of the AttnGAN network of the present invention, the generation loss and the discrimination loss are defined as:

[0072]

[0073] i is the stage value, i is 0 for the first stage, 1 for the second stage, and 2 for the third stage;

[0074] The unconditional loss is used to train to generate images that conform to the true data distribution to fool the discriminator, while the conditional loss is trained to generate sample images that match the text description.

[0075] Discriminator D i The training is done by minimizing the cross entropy loss to classify the input into real or fake images:

[0076]

[0077] I i is the distribution of real data samples at different stages I i is the distribution of generated data samples from the same stage

[0078] In order to generate semantically consistent and realistic images, the final objective function of the generator and discriminator is defined as:

[0079]

[0080] K represents the number of stages and is set to 2. i , λ0 and λ2 are hyperparameters that control different losses. The twin mechanism is used to improve the consistency of image generation. content Improving the discriminative ability of the discriminator from the word level guides the generator to generate images that are consistent with the given text description. In the loss, set α1=1, α2=2 and α3=3. DAMSM It is a measure of text-image matching score based on cosine similarity. Specifically, the Deep Attention Multimodal Similarity Model (DAMSM) in the AttnGAN model is used to construct a loss function to measure the image and text similarity at the word level in the text description and the corresponding image area. DAMSM loss makes the generated image more consistent with the text description.

[0081] In order to better measure the present invention, the present invention uses the public data sets Caltech-UCSD Birds (CUB) and COCO (Microsoft Common Objects in Context) to conduct experiments. The specific implementation effects are as follows: Figure 4 and Figure 5As shown. The present invention uses Inception Score (IS) to evaluate the quality and diversity of the generated images of the model, and adopts a pre-trained Inception-V3 network to calculate the KL divergence of the conditional distribution and the marginal distribution. However, IS cannot reflect the correlation between the generated image and the corresponding text description, so a novel evaluation index, the Visual-Semantic Similarity Score (VS), is used to measure the correlation between the generated image and the text. VS is an evaluation index that measures the content consistency of the image and text description. This index solves the alignment between the generated image and the conditional text. The higher the VS score, the higher the consistency between the image and the text.

[0082] It can be seen from the results in Table 1 that the present invention achieves higher IS in the CUB and COCO datasets, effectively improving the quality and diversity of generated images.

[0083] Table 1: IS index comparison table between the existing method and the image generated by the present invention

[0084]

[0085] In order to more clearly show the correlation between the generated image and the corresponding text words, the VS evaluation index is used for calculation, and the obtained results are shown in Table 2. The present invention effectively improves the consistency between the image and the corresponding text description.

[0086] Table 2: Comparison of VS indicators between existing methods and images generated by the present invention

[0087]

[0088] Embodiment 2:

[0089] Fréchet Inception Distance (FID) is a measure of the distance between the feature vectors of the real image and the generated image. The lower the FID score, the closer the generated image distribution is to the real image distribution. This embodiment uses FID to compare the FIDs of the SW-GAN model and other models; as shown in Table 3 below, the present invention has more advantages in learning the real image distribution.

[0090] Table 3: FID comparison of SW-GAN model and other models on CUB and COCO test datasets

[0091]

[0092] Embodiment 3:

[0093] In order to further verify the effectiveness of the twin mechanism module and word-level discrimination module proposed in the present invention, this embodiment continues to conduct ablation research on the CUB dataset, using IS and VS to evaluate the quality of each module in generating text-generated images. The measurement results are shown in Table 4. In the table, "—" indicates that the module is not used, and "√" indicates that the module is used. The effectiveness of each module of the present invention is verified through ablation research on different modules.

[0094] Table 4: IS and VS comparison of different modules on the CUB dataset

[0095]

[0096] The above description is only by way of illustration of certain exemplary embodiments of the present invention. It is undoubted that those skilled in the art can modify the described embodiments in various ways without departing from the spirit and scope of the present invention. Therefore, the above drawings and descriptions are illustrative in nature and should not be construed as limiting the scope of protection of the claims of the present invention.

Claims

1. A text-to-image generation method based on a twin-word-level discriminator, characterized in that: The steps include: S1: A multi-stage attention generative adversarial network is used as the basic network structure, including a text encoder, an image encoder, a generator, and a discriminator. The text encoder is a pre-trained bidirectional LSTM network that encodes the text description T, which consists of sentence features s and word features w. It generates images of different resolutions from coarse to fine in three stages. S2: Enhance the sentence feature s through a conditional enhancement technique, and then use the enhanced sentence feature F ca (s) is concatenated with the random vector z as the input of the first-stage generator G0; S3: In the second and third stage generators, an attention model is used to encode the word feature vector w and the hidden visual features n of the previous level generated graph i-1 Form an attention vector Fiattn(w,ni-1) and use them as the input of the corresponding generator to synthesize images of the corresponding resolution; S4: Add a twin discriminant module to the discriminator to learn the similarity between generated images and real images; S5: Generate image I according to the text description s , and randomly selected images that do not match the text description I s′ , using the twin mechanism to transform the positive sample into real I s and I s The distance between the two is shortened, and the negative sample I s′ and I s The distance is increased; the contrast loss is used to train the proposed twin structure; the contrast loss at each stage is defined as: d(I s ,I s ) 2 Represents the distance between the generated image and the real image after the twin mechanism. δ is used to identify whether the image is a positive sample or a negative sample, that is, 0 represents a negative sample and 1 represents a positive sample; S6: Use word-level discriminators to model the fine-grained associations between words in the text and the corresponding subregions of the image, and establish semantic-visual correlations between words and image subregions; The input of the level discriminator includes two aspects: text features and visual features.

2. The method for generating images from text based on a twin-word level discriminator according to claim 1, characterized in that: In step S6, for the text feature, the word feature w obtained from the text encoder, and sentence vector s, where w and The word features are respectively from the given text description T and the randomly selected word features that do not match the given text description T and are encoded; For visual features, the real image I obtained by encoding based on the image encoder s and generate images and I s ; Using p i,j Represents visual features I s and I s , w={w k |k=1,...,L} represents word level w and Features, calculate contextual features of different image regions and words in text descriptions: Where m k is the attention weight of the kth word, p i,j A feature vector representing a sub-region of the image; All word context correlations are summed to calculate the final correlation value between image I and sentence T: The context feature L content , visual features p i,j And the global sentence feature s is connected as the input of the discriminator to determine whether the input image is true or false.

3. The method for generating images from text based on a twin-word level discriminator according to claim 2, characterized in that: The generation loss and discrimination loss of the three stages in step S1 are defined as: i is the stage value, i is 0 for the first stage, 1 for the second stage, and 2 for the third stage; The unconditional loss is used to train the generation of images that match the real data distribution to fool the discriminator, while the conditional loss is trained to generate sample images that match the text description; Discriminator D i The training is done by minimizing the cross entropy loss to classify the input into real or fake images: I i is the distribution of real data samples at different stages I i is the distribution of generated data samples from the same stage 4. The method for generating images from text based on a twin-word level discriminator according to claim 3, characterized in that: The final objective function of the generator and discriminator is defined as: K represents the number of stages; α i , λ0 and λ2 are hyperparameters that control different losses; The twin mechanism is used to improve the consistency of image generation; content Improving the discriminative ability of the discriminator from the word level guides the generator to generate images that are consistent with the given text description; In the loss, we set α1=1, α2=2 and α3=3, L DAMSM It is a text-image matching score measured based on cosine similarity.

5. The method for generating images from text based on twin-word level discriminator according to claim 4, characterized in that: The image generation process of the generator in step S3 is as follows: n0=F0(z,F ca (s)); n i =F i (n i-1 ,F i attn (w,n i-1 )),i=1,2; I i =G i (n i ),i=0,1,2 Among them, {n0,n1,n2} represent the hidden layer features at different stages, and {I0,I1,I2} represent the generated images of different scales: 64×64, 128×128 and 256×256.

6. The method for generating images from text based on twin-word level discriminator according to claim 5, characterized in that: In step S5, an adjustment parameter is added to the contrast loss. and Adaptively adjust the weights of positive and negative sample pairs; the sliding loss is defined as: γ represents an adjustable sliding hyperparameter; when the sliding hyperparameter γ = 0, the sliding loss represents the loss; when γ > 0, the greater the distance between the positive sample pairs, the greater the penalty weight The bigger it is.

7. The method for generating images from text based on twin-word level discriminator according to claim 1, characterized in that: In step S1, a pre-trained Inception-V3 network is used as an image encoder.

8. The method for generating images from text based on twin-word level discriminator according to claim 1, characterized in that: The hyperparameters used in step S5 To balance the distance value when δ=0.

Images