A method for generating adversarial network discriminator and text-generated image

Abstract

The present invention discloses a generative adversarial network discriminator and a method for generating images from text. The discriminator is designed from three aspects. First, a diversity-sensitive conditional discriminator is proposed, which increases the diversity of generated images by judging the combination of generated images and mismatched text as false. Secondly, for the non-conditional discriminator, a comparative search strategy is proposed to measure the realism of the generated images, and the gradient is penalized to stabilize the training process. Finally, the discriminator is used as a feature extractor, and a multi-level image similarity loss is introduced to further improve the high-level feature similarity between the training images and the generated images and the targets. A large number of experimental results and ablation studies show that the improvement of the discriminator can effectively improve the quality of the generated images.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision and multimedia, and in particular to a method for generating an adversarial network discriminator and generating images from text. Background Art

[0002] Generating realistic images from text descriptions is an active research area in computer vision and multimedia. The goal is to generate realistic images that can show as much semantic information as possible from the text description. Generative Adversarial Networks (GAN) play an important role in text-to-image generation because of its generator-discriminator structure that is suitable for cross-modal transfer tasks.

[0003] Many works have focused on improving the generator and fine-grained text-image consistency. In practice, the discriminator is very important in providing correct guidance for the training of the generator. In the present invention, the ability of the discriminator in the GAN system is improved to provide correct guidance for the training of the generator. The earliest discriminator used for text-generated images was able to determine whether the features of the generated image were consistent with the sentence feature vector. Subsequently, HDGAN proposed a multi-layer nested multi-scale image discriminator to jointly train the image generator. StackGAN++ proposed a conditional discriminator and a non-conditional discriminator, which are used to determine whether the fake image matches the input description and distinguish the realism of the image. This bidirectional discriminator has been widely used in subsequent work.

[0004] Text-to-image generation is a multimodal task, where multiple images can visualize an input text. However, the bidirectional discriminator ignores this multimodal feature and imposes an overly strict penalty on multimodal tasks. Strict discriminators can lead to mode collapse and unstable training of GANs. In order to increase the diversity of generated images and alleviate the instability of GAN training, some researchers have proposed Hinge Loss to stabilize GAN training, but this may affect the authenticity of generated images and their similarity to training images. Other methods treat some generated images as real images during the discrimination process. However, this strategy uses a fixed number of real images as real images in each training step, which reduces the accuracy of the discriminator.

[0005] Improving the high-level semantic similarity between generated images and corresponding training images can further increase the diversity between generated images. Some recent works SEGAN and XMCGAN add image similarity loss to high-level image features to improve the similarity between the two images. For example, XMCGAN proposed contrast loss between images and text, regions and words, and generated images and training images to train the generator. This method also uses the discriminator network as an image feature extractor. However, this method does not specifically train the discriminator as a qualified feature extractor to distinguish high-level features of real and fake images, nor does it measure image similarity from multiple levels. Summary of the invention

[0006] In view of the shortcomings of the discriminator in the existing text generation method, the present invention designs a generative adversarial network discriminator and a method for generating images from text, which can alleviate the reduction in the diversity of generated images caused by mode collapse and make the network training process more stable, thereby improving the quality of generated images.

[0007] The generative adversarial network discriminator designed by the present invention is special in that it includes a diversity-sensitive conditional discriminator, a contrastive search gradient-penalized unconditional discriminator, and a multi-level image similarity unit, which respectively obtain conditional and unconditional adversarial losses and similarity losses between generated images and corresponding training images to train a better generator;

[0008] The diversity-sensitive conditional discriminator is based on the traditional conditional discriminator, and judges the combination of the generated image and the mismatched text as false, thereby generating a loss penalty to alleviate mode collapse;

[0009] The unconditional discriminator determines whether the generated image is real or fake by comparing the distance between the generated image and the constructed real image and fake image based on the gradient penalty strategy of contrast search, and can penalize the gradient explosion caused by underestimating the contrast of the real image;

[0010] The multi-level image similarity unit calculates the image-level difference and the object-level difference between the generated image and the corresponding training image, and adds the two difference losses to update the generator, thereby promoting the high-level semantic similarity between the generated image and the corresponding training image. Before calculating the multi-level image similarity loss, the discriminator network is first trained to enable it to reflect the difference between the real training image and the generated image, thereby helping to more accurately calculate the multi-level image similarity difference.

[0011] Furthermore, in the conditional discriminator, by changing the discriminant rules of the discriminator, the sensitivity of the traditional discriminator to the diversity of generated images is improved, thereby alleviating mode collapse and helping to generate more diverse images. The specific process is as follows:

[0012] In the traditional conditional discriminator, the features of the real training image or generated image are connected in series with the corresponding or non-corresponding text features. After the judgment of the discriminator, the combination judged as "false" is incurred with loss, which is used to train the generator. The traditional conditional discriminator generally makes judgments on three combinations of image and text features: the combination of the features of the real training image and the matching text features is judged as "true"; the combination of the features of the real training image and the features of other unmatched texts is judged as "false"; the combination of the features of the generated image and the matching text features is judged as "false". These three judgments are expressed in the following formulas:

[0013]

[0014] in, is the conditional discriminator, is a generator, is to generate an image, It is i A real training image, and is the sentence feature vector of the input text, is a noise vector sampled from a standard normal distribution, i = j Indicates that the image and text are matched. i ≠ j Indicates that the two do not match.

[0015] In the present invention, the third judgment in the above formula is changed to the following form, and the first and second judgments in the formula remain unchanged:

[0016]

[0017] That is, the conditional discriminator will judge the combination of the features of the generated image and the mismatched text features as "fake", thus changing the traditional conditional discriminator to a diversity-sensitive conditional discriminator, which will penalize generated images that do not match the input text. When mode collapse occurs, many generated images that should be different look very similar, so these generated images are often inconsistent with the corresponding input text semantics. The diversity-sensitive conditional discriminator will penalize this phenomenon and produce a more reasonable adversarial loss to train a better generator.

[0018] Furthermore, the unconditional discriminator determines whether the fake image is real or fake by comparing the distance between the generated image and the constructed real image and fake image. The specific process is as follows:

[0019] First, for a batch of generated images , build a batch of true graphs :

[0020]

[0021] in, is the training set image corresponding to this batch of generated images, The value of is randomly selected from [0.85,0.99]; similarly, a batch of constructed fake graphs are constructed :

[0022]

[0023] in, is a batch of noise images, which have the same resolution as the generated image, but each pixel is uniformly distributed obtained by sampling; The value of is randomly selected from [0.4,0.6]; for a generated image , to compare it with the corresponding true construction graph and construct fake graphs distance, thus determining Is it a true comparison image? First, input all three images into the unconditional discriminator. Then the two distances are calculated as follows:

[0024]

[0025] Among them, |·| represents the absolute value of the difference. Finally, if there is and , then it is believed that is a true comparison picture. In other cases, it is considered It's a fake comparison picture.

[0026] Furthermore, the hinge loss is calculated in the unconditional discriminator, and the resulting gradient explosion is penalized. The specific process is as follows:

[0027] First, the conventional hinge loss function is used to convert the current unconditional discriminator Updated to , and then use the following formula to further Updated to , to modify the original discriminator to judge that it is greater than The image will be judged by the updated discriminator as being smaller than :

[0028]

[0029] in, Will Updated to The loss function, E represents the expected value, Ncr is the value that the conventional discriminator judges to be greater than The discriminator after the image is updated Judged as less than Pictures, is the threshold for comparing the true image, represents the sigmoid function, The value of is the weight of the gradient penalty. The larger the value, the more images are generated. With higher realism, hyperparameters Used to adjust weight value.

[0030] Then, the unconditional discriminator is penalized with the updated contrastive search gradient The generated adversarial loss is used to train and update the generator:

[0031] Description: Description: Description: Description: Description: Description: Description: Description: Description: Description: Description: C:\Users\NOrth_\AppData\Roaming\Tencent\Users\2434132668\QQ\WinTemp\RichOle\}]@Z}QCC1CU23([GA0P~LBY.png

[0032] in It is an adaptive parameter whose value is a batch of real training image input In the above formula, let The value of and 1, which can reduce the difference between the training images and the generated images. is the noise vector, is the distribution of the noise vector” is added to the corresponding part of the invention content in the specification.

[0033] Furthermore, the specific process of the image level difference loss is as follows:

[0034] First, train the discriminator as an image feature extractor to construct positive and negative samples so that the real training images are less different from the positive samples and more different from the negative samples, as follows:

[0035] For a real picture , both positive and negative samples contain m pictures, where the positive samples contain high-fidelity Semantically consistent images, negative samples contain low-fidelity and Semantically inconsistent images will be Both positive and negative sample images are input into the similarity discriminator, and the output values ​​are , and , calculate and The value with the largest difference , and The smallest difference value , calculate the triplet loss Triplet Loss so that Less than :

[0036]

[0037] Where margin is a hyperparameter; then, the center loss is calculated to improve the discriminator network's ability to distinguish and robustness for positive and negative samples:

[0038]

[0039] in, yes and The average distance between

[0040] Then, the discriminator network trained with the above two losses is used to extract the image-level and object-level features of the generated images and the real training images, respectively, to calculate the multi-level image similarity loss, as follows:

[0041] Furthermore, the trained discriminator network is used to extract the features of the generated images and the features of the real training images respectively, and the difference between the two is calculated as the image-level similarity loss. The specific process is as follows:

[0042] The training images and generated images with a resolution of 256*256*3 are respectively input into the discriminator network shown in the figure, and 4*4*C intermediate features are extracted respectively, where C is the number of feature channels. Then, they are respectively passed through two convolutional layers to obtain the final discriminator output value. The two values ​​are subtracted to obtain the image-level difference value between the training image and the generated image.

[0043] Furthermore, the trained discriminator network is used to extract the features of the foreground objects in the generated images and the features of the foreground objects in the real training images, and the difference loss at the object level between the two is calculated. The specific process is as follows:

[0044] First, the YOLOv3 object detector is used to detect the foreground objects in the training images, and the bounding box is used to represent the position of the object. The 255-dimensional object features are extracted at each position. For the generated image, the position detected in the training image is used to extract the 255-dimensional features of the corresponding position of the generated image. For these two features, the average pooling layer is used to reduce the 255-dimensional features to 1-dimensional feature values, and then the absolute value difference of the feature values ​​of the corresponding objects in the two images is averaged to obtain the object-level difference value between the training image and the generated image.

[0045] Based on the same inventive concept, the present invention also designs a method for generating images from text using a generative adversarial network discriminator, comprising the following steps:

[0046] Step 1, input text information, use a text encoder to encode the text into a sentence feature vector and a word feature matrix;

[0047] Step 2: The encoded sentence feature vector is concatenated with a noise vector and input into the image generator to generate a preliminary image. The word feature matrix performs attention supervision on the preliminary image to generate a clearer image.

[0048] Step 3: Improve the traditional conditional discriminator into a diversity-sensitive conditional discriminator, input the real training images and generated images, and the constructed real and fake images into the diversity-sensitive conditional discriminator to generate a more reasonable adversarial loss to train the generator;

[0049] Step 4: For each generated image, a high-quality true image and a low-quality fake image are constructed, and the difference between the generated image and the two constructed images is compared. If the difference with the constructed true image is small, the generated image is judged as a comparison true image; if the difference with the constructed fake image is small, the generated image is judged as a comparison fake image, and the adversarial loss is generated to update the generator;

[0050] Step 5, calculate the hinge loss for the unconditional discriminator and preliminarily update the discriminator;

[0051] Step 6: The initially updated discriminator may have overfitting, that is, it may discriminate a part of the true image as false, resulting in a large gradient, which will cause instability in the training process. Gradient penalty is applied to this part of the gradient to stabilize the training process.

[0052] Step 7: For the training of the generator, the adversarial loss of Hinge Loss is calculated, and the generated image is made close to the value range of the real training image, so that the generated image is close to the real training image;

[0053] Step 8: Train the discriminator as an image feature extractor to construct positive samples and negative samples so that the real training images are less different from the positive samples and more different from the negative samples.

[0054] Step 9: Use the trained discriminator network to extract the features of the generated image and the features of the real training image, calculate the difference between the two as the loss function, and update the generator;

[0055] Step 10: Use the pre-trained YOLOv3 as an object detector to detect the similarity between objects in the real training image and objects in the corresponding positions in the generated image. The difference between the two is used as the loss function to update the generator.

[0056] Furthermore, the specific process of step 7 is as follows:

[0057] Penalize the unconditional discriminator with the updated contrastive search gradient The generated adversarial loss is used to train and update the generator:

[0058] Description: Description: Description: Description: Description: Description: Description: Description: Description: Description: Description: C:\Users\NOrth_\AppData\Roaming\Tencent\Users\2434132668\QQ\WinTemp\RichOle\}]@Z}QCC1CU23([GA0P~LBY.png

[0059] in It is an adaptive parameter whose value is a batch of real training image input In the above formula, let The value of and 1, which can reduce the difference between training images and generated images.

[0060] Preferably, the object detector adopts YOLOv3.

[0061] The advantages of the present invention are:

[0062] 1) By improving the traditional conditional discriminator, a diversity-sensitive conditional discriminator is proposed to increase the diversity of generated images and alleviate mode collapse;

[0063] 2) For the unconditional discriminator, an unconditional discriminator strategy based on contrastive search gradient penalty (CSGP) is proposed. This improvement can better evaluate the realism of the evaluation image and punish the gradient explosion phenomenon, thereby stabilizing the training of GAN;

[0064] 3) A multi-level image similarity loss is proposed to improve the similarity measurement method between the generated image and the corresponding training image. Before using the discriminator network to extract image and target features, the discriminator network is trained into a qualified feature extractor so that it can distinguish between good and bad image quality.

[0065] The present invention effectively improves the diversity and realism of generated images, improves the semantic similarity between generated images and real training images, and can stabilize the training process. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 It is an overall flow chart of the present invention for text-picture generation.

[0067] Figure 2 Schematic diagram of the non-conditional discriminator of contrastive search gradient penalty of the present invention.

[0068] Figure 3 Schematic diagram of multi-level image similarity loss of the present invention. DETAILED DESCRIPTION

[0069] The present invention is further described below by specific examples and drawings. The examples of the present invention are intended to enable those skilled in the art to better understand the present invention, and do not limit the present invention in any way.

[0070] The generative adversarial network discriminator designed by the present invention for text-generated image includes a conditional discriminator, an unconditional discriminator and a similarity discriminator, which can obtain conditional and unconditional adversarial losses and similarity losses between generated images and corresponding training images to train a better generator;

[0071] The diversity-sensitive conditional discriminator is based on the traditional conditional discriminator, and judges the combination of generated image features and mismatched text features as false, thereby generating a generator with better loss training, thereby alleviating the mode collapse of the generator and improving the diversity of generated images;

[0072] The unconditional discriminator determines whether the fake image is real or fake by comparing the distance between the generated image and the constructed real image and fake image based on the gradient penalty strategy of contrast search, and penalizes the gradient explosion caused by underestimating the contrast of the real image;

[0073] The multi-level image similarity unit calculates the image-level difference and the object-level difference, and adds these two difference losses to update the generator.

[0074] Based on the same inventive concept, the present invention designs a method for generating images from text using a generative adversarial network discriminator. The specific design process of the above discriminator is further described in detail in conjunction with the method for generating images from text. The specific process is as follows:

[0075] Step 1, input text information, use a text encoder to encode the text into a sentence feature vector and a word feature matrix;

[0076] Step 2: The encoded sentence feature vector is concatenated with a noise vector and input into the image generator to generate a preliminary image. The word feature matrix performs attention supervision on the preliminary image to generate a clearer image.

[0077] Step 3: Improve the traditional conditional discriminator to a diversity-sensitive conditional discriminator. Input the real training images and generated images, the constructed real images and fake images into the diversity-sensitive conditional discriminator to generate a more reasonable adversarial loss to train the generator. The details are as follows:

[0078] In the traditional conditional discriminator, the features of the real training image or generated image are connected in series with the corresponding or non-corresponding text features. After the judgment of the discriminator, the combination judged as "false" is incurred with loss, which is used to train the generator. The traditional conditional discriminator generally makes judgments on three combinations of image and text features: the combination of the features of the real training image and the matching text features is judged as "true"; the combination of the features of the real training image and the features of other unmatched texts is judged as "false"; the combination of the features of the generated image and the matching text features is judged as "false". These three judgments are expressed in the following formulas:

[0079]

[0080] in, is the conditional discriminator, is a generator, is to generate an image, It is i A real training image, and is the sentence feature vector of the input text, is a noise vector sampled from a standard normal distribution, i = j Indicates that the image and text are matched. i ≠ j Indicates that the two are not matched. In the present invention, the third judgment in the above formula is changed to the following form, and the first two judgments remain unchanged:

[0081]

[0082] The diversity-aware conditional discriminator has three advantages: first, generated images with poor realism are unlikely to match the corresponding text, which means that the third judgment of the traditional discriminator is rarely used. In most cases, the generated images are neither realistic nor semantically match the input text, especially in the early stage of GAN training; second, it is more reasonable to remove this judgment when the generated images have high quality in the later stage of GAN training. In practice, due to the multimodal nature of the text-to-image task, the generated qualified images will not look the same as the corresponding training images. When there are high-quality generated images that match the corresponding text but are different from the corresponding training images, the traditional conditional discriminator may become too strict. In addition, this overly strict conditional discriminator will further lead to mode collapse in image generation; the modified discriminator will penalize generated images that do not match the corresponding input text. When mode collapse occurs, many generated images are visually similar. These generated images are often inconsistent with the corresponding input text semantics. The modified conditional discriminator will penalize this phenomenon and produce a more reasonable adversarial loss to train a better generator. Therefore, the modified conditional discriminator can alleviate mode collapse and help generate more diverse images.

[0083] Step 4: For each generated image, construct a high-quality true image and a low-quality fake image, and compare the difference between the generated image and the two constructed images. If the difference with the constructed true image is small, the generated image is judged as a comparison true image. If the difference with the constructed fake image is small, the generated image is judged as a comparison fake image, and the adversarial loss is generated to update the generator, as follows:

[0084] First, for a batch of generated images , build a batch of true graphs :

[0085]

[0086] in, is the training set image corresponding to this batch of generated images, The value of is randomly selected from [0.85,0.99]. Similarly, a batch of constructed fake graphs are constructed :

[0087]

[0088] in, is a batch of noise images, which have the same resolution as the generated image, but each pixel is uniformly distributed obtained by sampling in . The value of is randomly selected from [0.4,0.6]. For a generated image , to compare it with the corresponding true construction graph and construct fake graphs distance, thus determining Is it a true comparison image? The calculation method of these two distances is as follows:

[0089]

[0090] Here, |·| represents the absolute value of the difference, is a non-conditional discriminator. Finally, if there is and , then it is believed that is a true comparison picture. In other cases, it is considered It is a fake comparison picture;

[0091] Step 5: Calculate the hinge loss for the unconditional discriminator and preliminarily update the discriminator as follows:

[0092] In the training t Step 1, first use the method in the previous step to get the generator Get a set of real images for comparison from a batch of generated images And a set of fake comparison pictures , and use the following conventional Hinge Loss function to convert the current unconditional discriminator Updated to :

[0093]

[0094] in, E represents the expected value, P data represents the distribution of real training images, x~P data that is x The distribution of follows the distribution of real training images. is to generate pictures, x These are real training pictures.

[0095] Step 6: The initially updated discriminator may be overfitting, that is, it may discriminate a part of the true image as false, resulting in a large gradient, which will cause instability in the training process. Gradient penalty is applied to this part of the gradient to stabilize the training process, as follows:

[0096] Conventional loss functions are generally prone to unstable losses. That is, for real images with higher authenticity, they will also produce larger losses. The images that were originally judged as greater than by the discriminator will be judged as less than by the updated discriminator. This error is corrected to reduce the gradient and further converted to Updated to Description: Description: Description: Description: Description: Description: Description: Description: Description: C:\Users\NOrth_\AppData\Roaming\Tencent\Users\2434132668\QQ\WinTemp\RichOle\A40A]1H2VI_AB]V17{G0SO2.png , is the threshold for comparing the true image, In this experiment, it is set to 0; the authenticity is determined by the comparison in step 4. The comparison true image obtained in the step is closer to the constructed true image, and its authenticity is considered to be higher, so it is called "the comparison true image with higher authenticity"; similarly, the comparison fake image is considered to have a lower authenticity.

[0097] Description: Description: Description: Description: Description: Description: Description: Description: Description: C:\Users\NOrth_\AppData\Roaming\Tencent\Users\2434132668\QQ\WinTemp\RichOle\7RBNX5TC]}%N{UQX0XV@OPJ.png

[0098] in, To describe: Description: Description: Description: Description: Description: Description: Description: Description: C:\Users\NOrth_\AppData\Roaming\Tencent\Users\2434132668\QQ\WinTemp\RichOle\OY4IW}Z$7J1Y0D]V~FTXBCM.png Updated to Description: Description: Description: Description: Description: Description: Description: Description: Description: C:\Users\NOrth_\AppData\Roaming\Tencent\Users\2434132668\QQ\WinTemp\RichOle\A40A]1H2VI_AB]V17{G0SO2.png The loss function is represents the sigmoid function, The value of is the weight of the gradient penalty. The larger the value, the more images are generated. Description: Description: Description: Description: Description: Description: Description: Description: Description: C:\Users\NOrth_\AppData\Roaming\Tencent\Users\2434132668\QQ\WinTemp\RichOle\49KC4]A8(CRAZS%@7BM%YXM.png With higher realism, hyperparameters Used to distinguish weight values. In this embodiment The value of is 3, N cr The conventional discriminator judges that it is greater than The discriminator after the image is updated Judged as less than .

[0099] Step 7: For the training of the generator, the adversarial loss of Hinge Loss is calculated, and the distribution of the generated images is made close to the real training images, so that the generated images are close to the real training images, as follows:

[0100] With the updated discriminator The generated adversarial loss is used to train and update the generator:

[0101] Description: Description: Description: Description: Description: Description: Description: Description: Description: Description: Description: C:\Users\NOrth_\AppData\Roaming\Tencent\Users\2434132668\QQ\WinTemp\RichOle\}]@Z}QCC1CU23([GA0P~LBY.png

[0102] in It is an adaptive parameter whose value is a batch of real training image input In the above formula, let The value of and 1, which can reduce the difference between the training images and the generated images. is the noise vector, is the distribution of the noise vector.

[0103] Step 8: Train the discriminator as an image feature extractor to construct positive and negative samples so that the real training images are less different from the positive samples and more different from the negative samples, as follows:

[0104] For a real picture , both positive and negative samples contain m pictures, where the positive samples contain high-fidelity Semantically consistent images, negative samples contain low-fidelity and Semantically inconsistent images. Both positive and negative sample images are input into Figure 3 In the discriminator network shown, the output values ​​are , and Calculate and The value with the largest difference , And description: Description: Description: Description: Description: Description: Description: Description: Description: Description: Description: C:\Users\NOrth_\AppData\Roaming\Tencent\Users\2434132668\QQ\WinTemp\RichOle\(YI]V(9ACD}AU]H`X[GF04M.png The smallest difference value , calculate the triplet loss Triplet Loss so that Less than :

[0105]

[0106] Margin is a hyperparameter, and its value is set to 0.1. Calculating the center loss can improve the discriminator network's ability to distinguish positive and negative samples and its robustness:

[0107]

[0108] in, yes and The average distance between .

[0109] Step 9: Use the trained discriminator network to extract the features of the generated image and the features of the real training image, calculate the difference between the two as the loss function, and update the generator, as follows:

[0110] like Figure 3 As shown in the figure, for the similarity between the training image and the generated image at the picture level, the training image and the generated image with a resolution of 256*256*3 are respectively input into the discriminator network shown in the figure, and the intermediate features of 4*4*C are respectively extracted, where C is the number of feature channels, and then the final discriminator output value is obtained by passing through two convolutional layers respectively. The two values ​​are subtracted to obtain the picture-level difference value between the training image and the generated image;

[0111] Step 10: Use the pre-trained YOLOv3 as an object detector to detect the similarity between the objects in the real training image and the objects in the corresponding positions in the generated image. The difference between the two is used as the loss function to update the generator, as follows:

[0112] like Figure 3As shown in the figure, for the similarity between the training image and the generated image at the object level, the YOLOv3 object detector is first used to detect the foreground objects in the training image, and the position of the object is represented by a bounding box. The 255-dimensional object features are extracted at each position. For the generated image, the position detected in the training image is used to extract the 255-dimensional features of the corresponding position in the generated image. For these two features, the average pooling layer is used to reduce the 255-dimensional features to 1-dimensional feature values, and then the absolute value difference of the feature values ​​of the corresponding objects in the two images is averaged to obtain the difference value at the object level between the training image and the generated image. The difference values ​​at these two levels are used as losses to train a better generator.

[0113] It should be understood that the embodiments and examples discussed here are for illustrative purposes only and may be modified or altered by those skilled in the art, and all such modifications and alterations should fall within the scope of protection of the appended claims of the present invention.

Claims

1. A generative adversarial network discriminator, characterized in that: It includes a diversity-sensitive conditional discriminator, a contrastive search gradient-penalized unconditional discriminator, and a multi-level image similarity unit to obtain conditional and unconditional adversarial losses as well as similarity losses between generated images and corresponding training images to train better generators. The diversity-sensitive conditional discriminator is based on the traditional conditional discriminator, which judges the combination of generated images and mismatched texts as false, thereby generating a loss penalty to alleviate mode collapse; The unconditional discriminator is based on the gradient penalty strategy of contrastive search. It determines whether the fake image is real or fake by comparing the distance between the generated image and the constructed true image and fake image, and penalizes the gradient explosion caused by underestimating the contrast of the true image. The specific process is as follows: First, for a batch of generated images , build a batch of true graphs : in, is the training set image corresponding to this batch of generated images, The value of is randomly selected from [0.85,0.99]; similarly, a batch of constructed fake graphs are constructed : in, is a batch of noise images, which have the same resolution as the generated image, but each pixel is uniformly distributed obtained by sampling; The value of is randomly selected from [0.4,0.6]; for a generated image , to compare it with the corresponding true construction graph and construct fake graphs distance, thus determining Is it a true comparison picture? The calculation method of these two distances is as follows: Here, |·| represents the absolute value of the difference, Is a non-conditional discriminator, finally, if there is and , then it is believed that is a true comparison picture. In other cases, it is considered It is a fake comparison picture; The multi-level image similarity unit calculates the image level difference and the object level difference, and adds these two difference losses to update the generator; Among them, the hinge loss is calculated in the unconditional discriminator, and the resulting gradient explosion is penalized. The specific process is as follows: First, use the conventional hinge loss function to convert the current unconditional discriminator Updated to , and then use the following formula to further Updated to , to modify the original discriminator to judge that it is greater than The image will be judged by the updated discriminator as less than : in, represents the sigmoid function, The value of is the weight of the gradient penalty, The larger the value, the more images are generated. With higher realism, hyperparameters Used to distinguish weight values.

2. The generative adversarial network discriminator according to claim 1, characterized in that: The specific process of the image-level difference loss is as follows: The discriminator is trained as an image feature extractor to construct positive and negative samples so that the real training images are less different from the positive samples and more different from the negative samples, as follows: For a real picture , both positive and negative samples contain m pictures, where the positive samples contain high-fidelity Semantically consistent images, negative samples contain low-fidelity and Semantically inconsistent images will Both positive and negative sample images are input into the similarity discriminator, and the output values ​​are , and , calculate and The value with the largest difference , and The smallest difference value , calculate the Triplet Loss so that Less than : Among them, margin is a hyperparameter, and Center Loss is calculated to improve the discriminator network's ability to distinguish and robustness for positive and negative samples: in, yes and The average distance between Use the trained discriminator network to extract the features of the generated images and the features of the real training images respectively, and calculate the difference between the two as the image-level difference loss.

3. The generative adversarial network discriminator according to claim 2, characterized in that: Use the trained discriminator network to extract the features of the generated image and the features of the real training image respectively, and calculate the difference between the two as the image-level difference loss. The specific process is as follows: The training images and generated images with a resolution of 256*256*3 are input into the discriminator network respectively, and 4*4*C intermediate features are extracted respectively, where C is the number of feature channels. Then, they are passed through two convolutional layers to obtain the final discriminator output value. The two values ​​are subtracted to obtain the image-level difference value between the training image and the generated image.

4. The generative adversarial network discriminator according to claim 1, characterized in that: The object-level difference loss is specifically: For the similarity between training images and generated images at the object level, we first use the YOLOv3 object detector to detect the foreground objects in the training images, and use bounding boxes to represent the positions of the objects. We extract 255-dimensional object features at each position. For the generated images, we use the positions detected in the training images to extract 255-dimensional features of the corresponding positions in the generated images. For these two features, the average pooling layer is used to reduce the 255-dimensional features to 1-dimensional feature values, and then the absolute value difference of the feature values ​​of the corresponding objects in the two images is averaged to obtain the object-level difference between the training image and the generated image.

5. A method for generating images from text using the generative adversarial network discriminator described in any one of claims 1 to 4, characterized in that: The following steps are involved: Step 1, input text information, use a text encoder to encode the text into a sentence feature vector and a word feature matrix; Step 2: The encoded sentence feature vector is connected in series with a noise vector and input into the image generation unit to generate a preliminary image. The word matrix is ​​input to perform attention supervision on the preliminary image to generate a clearer image. Step 3: Input the real training images, generated images, constructed real images and fake images into the diversity-sensitive conditional discriminator to generate a more reasonable adversarial loss; Step 4: For each generated image, a high-quality real image and a low-quality fake image are constructed. The contrast search gradient penalty unconditional discriminator first compares the difference between the generated image and the two constructed images through the distance between the images, and generates an adversarial loss to update the generator. First, for a batch of generated images , build a batch of true graphs : in, is the training set image corresponding to this batch of generated images, The value of is randomly selected from [0.85,0.99]; similarly, a batch of constructed fake graphs are constructed : in, is a batch of noise images, which have the same resolution as the generated image, but each pixel is uniformly distributed obtained by sampling; The value of is randomly selected from [0.4,0.6]; for a generated image , to compare it with the corresponding true construction graph and construct fake graphs distance, thus determining Is it a true comparison picture? The calculation method of these two distances is as follows: Here, |·| represents the absolute value of the difference, Is a non-conditional discriminator, finally, if there is and , then it is believed that is a true comparison picture. In other cases, it is considered It is a fake comparison picture; Step 5: For the contrastive search gradient penalty non-conditional discriminator, calculate Hinge Loss and preliminarily update the discriminator. The specific process is as follows: First, use the conventional hinge loss function to convert the current unconditional discriminator Updated to , and then use the following formula to further Updated to , to modify the original discriminator to judge that it is greater than The image will be judged by the updated discriminator as less than : in, represents the sigmoid function, The value of is the weight of the gradient penalty, The larger the value, the more images are generated. With higher realism, hyperparameters Used to distinguish weight values; Step 6: Perform gradient penalty on the initially updated contrast search gradient penalty non-conditional discriminator to eliminate overfitting, thereby stabilizing the training process. Step 7: For the training of the generator, the adversarial loss of Hinge Loss is calculated, and the generated image is made close to the value range of the real training image, so that the generated image is close to the real training image; Step 8: Train a similarity discriminator as an image feature extractor to construct positive samples and negative samples, so that the real training images are less different from the positive samples and more different from the negative samples. Step 9: Use the trained similarity discriminator to extract the features of the generated image and the features of the real training image, calculate the difference between the two as the image-level difference loss function, and update the generator; Step 10: Use the pre-trained object detector to detect the similarity between the objects in the real training image and the objects in the corresponding positions in the generated image. The difference between the two is used as the object-level difference loss function to update the generator.

6. The method for generating an image from text according to claim 5, characterized in that: The specific process of step 7 is as follows: Penalize the unconditional discriminator with the updated contrastive search gradient The generated adversarial loss is used to train and update the generator: in It is an adaptive parameter whose value is a batch of real training image input The minimum value of The value of and 1, which can reduce the difference between training images and generated images.

7. The method for generating an image from text according to claim 6, characterized in that: The object detector adopts YOLOv3.

Images