A method to reduce mode collapse in generative adversarial networks

Abstract

The present invention discloses a method for reducing the mode collapse of a generative adversarial network, which includes 1) constructing a GMN-GAN network, wherein the GMN-GAN network includes a Gaussian mixture model M, a Gaussian mixture noise variable generator V, a sample generator G and a discriminator D; the number of Gaussian mixture noise variable generators is equal to the number of image categories c in an image data set, the output of the Gaussian mixture noise variable generator is used as the input of the sample generator, the output of the sample generator is used as the input of the discriminator, the number of output units of the discriminator output layer is set to the number of image categories, and one output unit corresponds to a different category; 2) training the GMN-GAN network. The sample generator in the present invention can generate more types of images, which can effectively avoid the problem of mode collapse in the generative adversarial network.

Description

Technical Field

[0001] The present invention relates to the field of neural network technology, and more particularly to a method for reducing the collapse of generative adversarial network modes. Background Art

[0002] In the past decade, image generation technology has played a key role in different fields, such as visual recognition, speech recognition and generation, and natural language processing. Well-known generative models generally include likelihood-based methods, generative adversarial networks (GANs), and diffusion-based networks.

[0003] GAN consists of two models, a generative model G and a distribution model D. Through the game of competition between G and D, G is gradually induced to generate images to deceive D. In recent years, as GAN has become more and more popular, some scholars have proposed methods that can generate high-definition images, and even the human eye cannot detect the difference between these images and real-world images. Generative adversarial networks also perform better in most image generation tasks than likelihood-based methods and diffusion-based networks.

[0004] Despite the great success of GANs in generating realistic images, it is well known that one of the most difficult problems to deal with in training GAN models is mode collapse. Mode collapse refers to the fact that the generator may collapse to a certain setting and always produce the same output. The regularity theorem of the optimal transport map is used to explain mode collapse: the root cause of mode collapse is the conflict between the regularity of the transport map and the continuous function space of deep neural networks (DNNs). Since DNNs can only model continuous mappings, and the mappings required for real images are not continuous, the main difficulty of GANs lies in the existence of this internal conflict. When using GANs to search for discontinuous mappings in the continuous mapping space, this search operation will not converge, or it will converge to a continuous branch of the target mapping, which will eventually lead to mode collapse. Summary of the invention

[0005] In view of this, an object of the present invention is to provide a method for reducing the mode collapse of a generative adversarial network, so as to solve the technical problem of how to reduce the mode collapse of a generative adversarial network when using a generative adversarial network to generate images.

[0006] The method of reducing the collapse of the generative adversarial network mode of the present invention comprises the following steps:

[0007] 1) constructing a GMN-GAN network, wherein the GMN-GAN network includes a Gaussian mixture model M, a Gaussian mixture noise variable generator V, a sample generator G, and a discriminator D;

[0008] The number of Gaussian mixture noise variable generators is equal to the number of image categories c in the image data set. The i-th Gaussian mixture noise variable generator V i The i-th image in the image data set is generated by fitting the Gaussian mixture model M, 1≤i≤c; the output of the Gaussian mixture noise variable generator is used as the input of the sample generator, and the output of the sample generator is used as the input of the discriminator. The number of output units in the output layer of the discriminator is set to the number of image categories, and one output unit corresponds to a different category;

[0009] The probability density function of the Gaussian mixture model M is defined as follows:

[0010]

[0011] Among them, x is the image sample dataset, f(x|μ k ,∑ k ) is the kth component, characterized by having a mean vector μ k and the covariance matrix ∑ k is a multivariate Gaussian distribution of , where K is the total number of components, and w k is the mixing weight and satisfies the equation Covariance matrix∑ k Can be a full-valued matrix or a diagonal matrix;

[0012] The loss function of the sample generator G is defined as follows:

[0013]

[0014] where y i represents the label corresponding to the i-th image sample and has been processed by one-hot encoding, m represents the number of samples, Represents and labels y i The corresponding i-th Gaussian mixture noise variable generator, Indicated by the label y i The corresponding noise variable generated by the i-th Gaussian mixture noise variable generator, represents the output of the noise after passing through the sample generator G, Represents the output of the sample generated by the sample generator after passing through the discriminator D;

[0015] The loss function of the discriminator D is defined as follows:

[0016]

[0017] Where D(x i ) represents the output of the real sample data used to train the network after passing through the discriminator D, where x i is an image sample dataset A real image sample in , n is the number of sample data, d is the dimension of sample data; the label corresponding to the image sample dataset X after one-hot encoding is Y,

[0018] The sample generator G and discriminator D pass the following loss function Play a minimax game:

[0019]

[0020] Where p r is the distribution of the real sample data used to train the network, p z is the distribution of fake sample data generated by the sample generator G;

[0021] 2) Training the GMN-GAN network includes the following steps:

[0022] a) Sample a batch of data X from the original training set, with label Y; select the corresponding Gaussian mixture noise variable generator V according to label Y to generate Gaussian mixture noise variable Z, and input Gaussian mixture noise variable Z into sample generator G to generate a batch of fake data X f ;

[0023] in:

[0024]

[0025] Where V i ∈M, M is a Gaussian mixture model; z(V i ) represents V i The generated Gaussian mixture noise variable, j is the dimension of the Gaussian mixture noise variable;

[0026] b) Input the real data X into the discriminator D to obtain the output vector D of the last layer of the neural network x , and compare it with the one-hot encoded true label Y valid Input into the loss function of the discriminator D to calculate part of the loss D_loss_1 of D; input the Gaussian mixed noise variable Z into the discriminator D to obtain the output vector D z , and compare it with the label Y that is judged as false fakeInput it into the loss function of the discriminator D to calculate the other part of the loss D_loss_2, and take the sum of the two parts of the loss (D_loss_1+D_loss_2) as the overall loss of the discriminator D, and then update the parameter weights of the discriminator D through gradient back propagation;

[0027] c) Input the Gaussian mixed noise variable Z into the discriminator D to obtain the output vector D z , and compare it with the true label Y valid Input into the loss function of the sample generator G to calculate its loss G_loss, and then update the parameter weights of G through gradient back propagation;

[0028] d) Repeat steps a), b), and c) until the preset number of iterations is reached to complete the training of the sample generator G.

[0029] Beneficial effects of the present invention:

[0030] The method of reducing the mode collapse of the generative adversarial network of the present invention is that each Gaussian mixture noise variable generator is generated by fitting a Gaussian mixture model, so that the distribution of various types of real images can be fitted more carefully, and then these Gaussian mixture noise variable generators are used to generate various random Gaussian mixture noise variables. The sample generator uses a multi-mode Gaussian mixture noise variable as input, rather than using a single distribution of noise variables as the input of the sample generator, which enhances the diversified information of the sample. And after fitting the Gaussian mixture model, the generated Gaussian mixture noise variable is multi-peak continuous distribution, which is significantly different from the existing image input of the generative adversarial network, which is discrete. This can effectively avoid the mode collapse of the generative adversarial network.

[0031] At the same time, in order to force the Gaussian mixture noise variable generator to capture the diverse information of real samples, this method one-hot encodes the original labels of real samples and fake samples, and the traditional binary discriminator becomes a multi-classification discriminator. In this way, the discriminator is not only trained to distinguish fake samples from real samples, but also to classify real samples. At the same time, the generator will update the prediction error of each type of fake samples according to the discriminator.

[0032] Based on this design, the generator can generate a wider variety of images and effectively avoid the problem of mode collapse in the generative adversarial network. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 This is the framework diagram of the GMN-GAN network;

[0034] Figure 2 Generate schematic diagrams for Gaussian mixture noise variables;

[0035] Figure 3 This is the GMN-GAN network training flow chart;

[0036] Figure 4 The pictures generated by the sample generator in different training iterations (the upper row of pictures are the images generated by GMN-GAN, and the lower row of pictures are the images generated by GAN);

[0037] Figure 5 The relationship diagram of KL divergence on GAN variants and GMN-GAN respectively;

[0038] Figure 6 The relationship diagram of Wasserstein distance on GAN variant and GMN-GAN respectively;

[0039] Figure 7 The loss change diagram of the sample generator and discriminator of GAN in iterative training;

[0040] Figure 8 The loss change diagram of the sample generator and discriminator of GMN-GAN in iterative training;

[0041] Fig. 9 Fashion-MNIST images generated for GMN-DCGAN. DETAILED DESCRIPTION

[0042] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0043] The method for reducing the collapse of the generative adversarial network mode in this embodiment includes the following steps:

[0044] 1) Construct a GMN-GAN network, which includes a Gaussian mixture model M, a Gaussian mixture noise variable generator V, a sample generator G and a discriminator D.

[0045] The number of Gaussian mixture noise variable generators is equal to the number of image categories c in the image data set. The i-th Gaussian mixture noise variable generator V i The i-th image in the image dataset is generated by fitting the Gaussian mixture model M, 1≤i≤c; the output of the Gaussian mixture noise variable generator is used as the input of the sample generator, and the output of the sample generator is used as the input of the discriminator. The number of output units of the discriminator output layer is set to the number of image categories, and one output unit corresponds to a different category.

[0046] The probability density function of the Gaussian mixture model M is defined as follows:

[0047]

[0048] Where x is the image sample dataset, f(x|μ k ,∑ k ) is the kth component, characterized by having a mean vector μ k and the covariance matrix ∑ k is a multivariate Gaussian distribution of , where K is the total number of components, and w k is the mixing weight and satisfies the equation Covariance matrix∑ k Can be a full-valued matrix or a diagonal matrix;

[0049] The loss function of the sample generator G is defined as follows:

[0050]

[0051] where y i represents the label corresponding to the i-th image sample and has been processed by one-hot encoding, m represents the number of samples, Represents and labels y i The corresponding i-th Gaussian mixture noise variable generator, Indicated by the label y i The corresponding noise variable generated by the i-th Gaussian mixture noise variable generator, represents the output of the noise after passing through the sample generator G, Represents the output of the sample generated by the sample generator after passing through the discriminator D.

[0052] The loss function of the discriminator D is defined as follows:

[0053]

[0054] Where D(x i ) represents the output of the real sample data used to train the network after passing through the discriminator D, where x i is an image sample dataset A real image sample in , n is the number of sample data, d is the dimension of sample data; the label corresponding to the image sample dataset X after one-hot encoding is Y,

[0055] First, the label of the real image is one-hot encoded. The position corresponding to the class in the label is marked as 1, and the rest are 0, that is, the image belongs to a certain class, and the label corresponding to the real image is called the real label; if all the positions of the label are marked as 0, the label is called a false label. Then, the number of output units of the discriminator is set to n, where n is the number of categories. After each image passes through the discriminator, a one-dimensional vector with n values ​​will be obtained. The discriminator updates the weights according to the prediction error of each class of real samples and the prediction error of false samples. The sample generator updates the weights according to the difference between the discriminator's predicted label for the false sample and the real label corresponding to the real sample.

[0056] The sample generator G and the discriminator D pass the following loss function Play a minimax game:

[0057]

[0058] 2) Training the GMN-GAN network includes the following steps:

[0059] a) Sample a batch of data X from the original training set, with label Y; select the corresponding Gaussian mixture noise variable generator V according to label Y to generate Gaussian mixture noise variable Z, and input Gaussian mixture noise variable Z into sample generator G to generate a batch of fake data X f ;

[0060] in:

[0061]

[0062] Where V i ∈M, M is a Gaussian mixture model; z(V i ) represents V i The generated Gaussian mixture noise variable, j is the dimension of the Gaussian mixture noise variable.

[0063] b) Input the real data X into the discriminator D to obtain the output vector D of the last layer of the neural network x , and compare it with the one-hot encoded true label Y valid Input into the loss function of the discriminator D to calculate part of the loss D_loss_1 of D; input the Gaussian mixed noise variable Z into the discriminator D to obtain the output vector D z , and compare it with the label Y that is judged as false fake Input it into the loss function of the discriminator D to calculate the other part of the loss D_loss_2, and take the sum of the two parts of the loss (D_loss_1+D_loss_2) as the overall loss of the discriminator D. After that, the parameter weights of the discriminator D are updated through gradient back propagation.

[0064] c) Input the Gaussian mixed noise variable Z into the discriminator D to obtain the output vector D z , and compare it with the true label Y valid The input is into the loss function of the sample generator G to calculate its loss G_loss, and then the parameter weights of G are updated through gradient back propagation.

[0065] d) Repeat steps a), b), and c) until the preset number of iterations is reached to complete the training of the sample generator G.

[0066] The following experiments are conducted to verify the effectiveness of the method of reducing the collapse of the GAN mode in this embodiment.

[0067] Dataset Information

[0068] The experiment selected MNIST, Fashion-MNIST, and CIFAR-10 datasets for evaluation. MNIST and FashionMNIST are small grayscale image datasets, including 60,000 training images and 10,000 test images, which are grayscale handwritten digital images and clothing images of size 28×28, respectively. CIFAR-10 includes ten unique categories: airplane, car, bird, cat, deer, dog, frog, horse, ship, and truck. The detailed information of the three datasets is shown in Table 1.

[0069] Table 1 Dataset details

[0070]

[0071] Evaluation indicators and selection models

[0072] In the experiment, the Inception Score (IS) and Fréchet Inception Distance (FID) were used as evaluation indicators for generated images. IS is an evaluation indicator for the quality and diversity of generated images. A larger IS value indicates that the sample generator G performs better. However, IS does not utilize the statistics of real-world samples, nor does it compare the statistics of the tested images with other generated samples. Unlike IS, FID takes into account the statistics of real-world samples, and it can better capture the similarity between generated images and real images than IS. Smaller FID values ​​have been shown to generate higher quality images. Due to different applicable backgrounds, the experiment matched the appropriate evaluation indicators for the dataset.

[0073] KL divergence (Kullback-Leibler divergence) is a widely used tool in statistics and pattern recognition to measure the distance between two probability density functions p(x) and q(x):

[0074]

[0075] The Wasserstein distance is a measure of the minimum effort required to reconfigure the probability mass of one distribution to recover the other distribution. Therefore, the KL divergence plot and the Wasserstein distance plot can intuitively observe the distance between two images.

[0076] In terms of model selection, in order to illustrate that the proposed GMN-GAN network can be applied to general GAN ​​variant networks, the experiment selected the classic WGAN-GP and DCGAN with convolution as the basic network architecture. WGAN-GP is proposed to cut weights, which penalizes the gradient norm of the discriminator relative to its input (generated fake images). DCGAN is a topological constraint based on a convolutional structure. The change of constraints enables it to be stably trained in most environments, especially for complex datasets such as CIFAR-10.

[0077] Experimental details

[0078] Experimental results are used to verify whether the design of GMN-GAN can alleviate the mode collapse problem. Specifically, experiments are conducted on the MNIST dataset on GAN, DCGAN and GMN-GAN. Figure 4 The following figure shows the images generated by the sample generator in different training iterations. As can be seen from the figure, GMN-GAN generates more image categories than the traditional GAN ​​(Vanilla GAN).

[0079] Figure 5 and Figure 6 The relationship between KL divergence and Wasserstein distance on GAN variants and GMN-GAN is shown respectively. Given the target distribution, KL divergence and Wasserstein distance can be used to calculate the distance between the real point and the generated point. The graphs of KL divergence and Wasserstein distance show that GMN-GAN has better scores than other GAN variants.

[0080] Figure 7 and Figure 8 The loss changes of the sample generator and discriminator of GAN and GMN-GAN in iterative training are shown respectively. From the ordinate in the figure, it can be seen that the loss of the sample generator and discriminator of GMN-GAN are smaller than that of GAN, so the method disclosed in this embodiment also stabilizes the training of the generative adversarial network to a certain extent.

[0081] Since applying the IS metric to generative models trained on datasets other than ImageNet can produce misleading results, in the experiment, IS is used to test images generated based on Cifar-10 (images come from a part of ImageNet data). Table 2 shows the results of the starting score, and it is noted that the introduction of GMN-GAN brings performance improvements in all experiments.

[0082] Table 2 Results of IS test on generating CIFAR-10 images

[0083]

[0084]

[0085] For the FID test, the data sets used in the embodiment are MNIST, Fashion-MNIST and CIFAR-10. Table 3 shows that the method proposed in this embodiment has the best FID value among all the tested methods. Fig. 9 The results of generating Fashion-MNIST images based on GMN-DCGAN are shown.

[0086] Table 3 Experimental FID score results of grayscale image dataset

[0087]

[0088] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solutions of the present invention, which should be included in the scope of the claims of the present invention.

Claims

1. A method for reducing mode collapse in a generative adversarial network, characterized by: The following steps are involved: 1) constructing a GMN-GAN network, wherein the GMN-GAN network includes a Gaussian mixture model M, a Gaussian mixture noise variable generator V, a sample generator G, and a discriminator D; The number of Gaussian mixture noise variable generators is equal to the number of image categories c in the image data set. The i-th Gaussian mixture noise variable generator V i The i-th image in the image data set is generated by fitting the Gaussian mixture model M, 1≤i≤c; the output of the Gaussian mixture noise variable generator is used as the input of the sample generator, and the output of the sample generator is used as the input of the discriminator. The number of output units in the output layer of the discriminator is set to the number of image categories, and one output unit corresponds to a different category; The probability density function of the Gaussian mixture model M is defined as follows: Among them, x is the image sample dataset, f(x|μ k ,∑ k ) is the kth component, characterized by having a mean vector μ k and the covariance matrix ∑ k is a multivariate Gaussian distribution of , where K is the total number of components, and w k is the mixing weight and satisfies the equation Covariance matrix∑ k is a full-valued matrix or a diagonal matrix; The loss function of the sample generator G is defined as follows: where y i represents the label corresponding to the i-th image sample and has been processed by one-hot encoding, m represents the number of samples, Represents and labels y i The corresponding i-th Gaussian mixture noise variable generator, Indicated by the label y i The corresponding noise variable generated by the i-th Gaussian mixture noise variable generator, represents the output of the noise after passing through the sample generator G, Represents the output of the sample generated by the sample generator after passing through the discriminator D; The loss function of the discriminator D is defined as follows: Where D(x i ) represents the output of the real sample data used to train the network after passing through the discriminator D, where x i is an image sample dataset A real image sample in , n is the number of sample data, d is the dimension of sample data; the label corresponding to the image sample dataset X after one-hot encoding is Y, The sample generator G and discriminator D pass the following loss function Play a minimax game: Where p r is the distribution of the real sample data used to train the network, p z is the distribution of fake sample data generated by the sample generator G; 2) Training the GMN-GAN network includes the following steps: a) Sample a batch of image data X from the original training set, with label Y; select the corresponding Gaussian mixed noise variable generator V according to label Y to generate Gaussian mixed noise variable Z, and input the Gaussian mixed noise variable Z into the sample generator G to generate a batch of fake data X f ; in: Where V i ∈M, M is a Gaussian mixture model; z(V i ) represents V i The generated Gaussian mixture noise variable, j is the dimension of the Gaussian mixture noise variable; b) Input the real data X into the discriminator D to obtain the output vector D of the last layer of the neural network x , and compare it with the one-hot encoded true label Y valid Input into the loss function of the discriminator D to calculate part of the loss D_loss_1 of D; input the Gaussian mixed noise variable Z into the discriminator D to obtain the output vector D z , and compare it with the label Y that is judged as false fake Input it into the loss function of the discriminator D to calculate the other part of the loss D_loss_2, and take the sum of the two parts of the loss (D_loss_1+D_loss_2) as the overall loss of the discriminator D, and then update the parameter weights of the discriminator D through gradient back propagation; c) Input the Gaussian mixed noise variable Z into the discriminator D to obtain the output vector D z , and compare it with the true label Y valid Input into the loss function of the sample generator G to calculate its loss G_loss, and then update the parameter weights of G through gradient back propagation; d) Repeat steps a), b), and c) until the preset number of iterations is reached to complete the training of the sample generator G.

Images