A GAN model improvement method for small data set scenarios

Abstract

The application belongs to the field of data enhancement application, and particularly relates to a GAN model improvement method for small data set scenarios. The multi-discriminator fusion module is composed of multiple discriminators and a discriminator fusion submodule. Each discriminator is a network composed of multiple convolutional layers and fully connected layers, which receives a batch of pictures in a downsampling manner and converts the pictures into a tensor. The tensor has four dimensions, namely the number of batches of images batch, the number of channels c, the width w of each image and the height h of each image. The multi-discriminator fusion submodule is responsible for coordinating multiple discriminators, receiving the return values of the multiple discriminators, weighting the return values and then transmitting the weighted return values to the generator for back propagation. In order to avoid the occurrence of sample penetration, the weight of the discriminator responsible for enhancement is reduced. In this way, the probability of overfitting of the discriminator can be reduced, and the possibility of sample penetration can be reduced.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of data enhancement application, and particularly relates to a GAN model improvement method for small data set scenarios. BACKGROUND

[0002] For the problem that GAN is difficult to effectively learn the parameter weights in the network model through limited data, data enhancement can overcome this difficulty to a certain extent. Unlike transfer learning which reduces the number of training parameters required by GAN, data enhancement provides effective information for the training of GAN by expanding the data set.

[0003] In recent years, data enhancement has made effective progress in improving GAN. Zhang et al. proposed a deep adversarial data augmentation (DADA) method in 2019. The method proposes a new discriminator loss function to realize data enhancement. Through the improved discriminator loss function, the real samples and the enhanced samples are calculated into the loss function to help and consistently find the decision boundary, thereby helping the GAN network training of small samples. Shaham et al. proposed SinGAN, which is an unconditional generative model. The model can learn features from a single natural image. After training, it can capture the internal distribution of the image and generate high-quality, diversified samples with the same visual content as the original image. SinGAN adopts a fully convolutional GAN, which contains multiple GANs, each of which is responsible for learning the image feature distribution at different scales. In this way, SinGAN can generate new samples with arbitrary size and aspect ratio while keeping the global structure and fine texture of the training image unchanged.

[0004] Although data augmentation has a significant effect on the training of GAN, researchers have proposed many robust models, but Karras et al. pointed out that in the process of data augmentation, the original data set may be contaminated, and the sample penetration problem may occur. Specifically, the data set input into the GAN not only contains the original data, but also contains the data after augmentation, and the data after augmentation is ultimately obtained by transforming the original data, and there is a certain difference between the two, and the augmented data cannot be completely equivalent to the real data. However, GAN treats the two as completely equivalent and treats them as real data. If the mixed samples are directly trained, the enhanced transformation changes the target distribution represented by the original training samples, which will inevitably make the sample distribution generated by the generator deviate from the real sample distribution. In some cases, this change will not make the generated false image deviate too much from the real image, but in some cases, this change will make the generated false image completely unusable. More fatally, it is uncontrollable in which case the generated false image is unusable, and this disadvantage greatly hinders the application of data augmentation in GAN. SUMMARY

[0005] In order to solve the problems in the background art, the application provides a GAN model improvement method for small data set scenarios.

[0006] The technical scheme of the application is as follows:

[0007] A GAN model improvement method for small data set scenarios, the steps are as follows:

[0008] Step 1: input a random vector z conforming to a Gaussian distribution into a generator G, and the generator G generates a false image G(z) according to the input random vector.

[0009] Step 2: input the false image G(z) generated by the generator G into a discriminator fusion module for gradient update.

[0010] The structure of the discriminator fusion module is as follows:

[0011] The discriminator fusion module includes an original discriminator and an enhanced discriminator. The training data set of the enhanced discriminator is different from that of the original discriminator, but the discriminator will still judge the image generated by the generator and feed back the judgment result to the generator. The target function of the improved multi-discriminator is as follows:

[0012] maxD Final =W D maxD+W D_Aug maxD Aug

[0013] Wherein, DFinal Wori is the weight of the original discriminator output value feedback to the generator, and Waug is the weight of the enhanced discriminator output value feedback to the generator. D D_Aug D D_Aug The sum of Wori and Waug is 1. In order to make the data distribution of the discriminator close to the real data set, Wori should be greater than Waug, and the value in the present application is Wori = 0.8, Waug = 0.2. That is, the weight of the discriminator inputting the real data set feedback to the generator is 80%, and the weight of the discriminator inputting the enhanced data set feedback to the generator is 20%, which can effectively reduce the occurrence of sample penetration problem. D D_Aug D D_Aug The sum of Wori and Waug is 1. In order to make the data distribution of the discriminator close to the real data set, Wori should be greater than Waug, and the value in the present application is Wori = 0.8, Waug = 0.2. That is, the weight of the discriminator inputting the real data set feedback to the generator is 80%, and the weight of the discriminator inputting the enhanced data set feedback to the generator is 20%, which can effectively reduce the occurrence of sample penetration problem.

[0014] In the single discriminator, all image data are first subjected to two-dimensional convolution layer for feature extraction, and then input into multi-layer residual block to prevent neural network overfitting, and dropout layer is added between every two residual blocks to reduce the overfitting ability of the discriminator. Then, the output result is subjected to pooling layer for down-sampling, the above operation is repeated for several times, and finally full connection layer is used for output of the result.

[0015] Step 3: After the discriminator fusion module receives the false image G(z) from the generator G, the original real image Xori and the false image G(z) are input into the original discriminator Dori for identification. At the same time, the enhanced real image Xaug and the false image G(z) are input into the enhanced discriminator Daug for identification.

[0016] Step 4: The loss values output by the original discriminator Dori and the enhanced discriminator Daug are weighted and mixed.

[0017] Step 5: The mixed result value is fed back to the generator G for gradient update.

[0018] Optimization of objective function: the objective function of the improved discriminator has not changed much compared with the original discriminator, only the discriminator is divided into two identical individuals for calculation, and then integrated, as shown below:

[0019]

[0020] In the model, the binary cross-entropy loss function is selected as the loss function, and its original calculation formula is as follows:

[0021]

[0022] ​​​​​​Since the loss function adopted is binary cross entropy, the loss function of the improved generator can be expressed as:

[0023]

[0024] The advantages of the present application include the following:

[0025] (1) The design of the multi-discriminator fusion module: the module is composed of multiple discriminators and a discriminator fusion submodule. Each discriminator is a network composed of multiple convolutional layers and fully connected layers, which receives a batch of pictures in a down-sampling manner and converts them into tensors. The tensor has four dimensions, namely the number of batches of images batch, the number of channels c, the width w of each image, and the height h of each image. Each image is labeled as true or false, and the false image is generated by the generator. The multi-discriminator fusion submodule is responsible for coordinating multiple discriminators, receiving the return values of multiple discriminators, and transmitting them to the generator after weighting for back propagation. Multiple discriminators will be responsible for the discrimination of the original data set and the enhanced data set. In order to avoid the occurrence of sample penetration, the weight of the enhanced discriminator will be reduced. In this way, the probability of overfitting of the discriminator can be reduced, and the possibility of sample penetration can be reduced.

[0026] (2) Optimization of the objective function: the objective function of the improved discriminator has not changed much compared with the original discriminator, only the discriminator is divided into two identical individuals for calculation, and then integrated. BRIEF DESCRIPTION OF DRAWINGS

[0027] Figure 1 is the overall framework diagram of the improved GAN model in embodiment 1;

[0028] Figure 2 is the framework diagram of the multi-discriminator module in embodiment 1;

[0029] Figure 3 is the schematic diagram of the original discriminator structure;

[0030] Figure 4 is the generator training process;

[0031] Figure 5 The left is the image before data enhancement processing, and the right is the image after data enhancement;

[0032] Figure 6 is the training process of AdaFM and MDGAN on the Oxflower102 data set in embodiment 1;

[0033] Figure 7 is the training process of AdaFM and MDGAN on the Flowers-25 data set in embodiment 2;

[0034] Figure 8 The method of AdaFM and MDGAN in Example 2 is trained on the Flowers-25 dataset to compare the generated images. DETAILED DESCRIPTION

[0035] The application will be described in detail below with reference to the accompanying drawings and examples:

[0036] As shown in Figures 1-5 , the application realizes the alleviation of the sample penetration problem that occurs after the data augmentation of the GAN dataset in the small dataset scenario. The main contributions of this method can be summarized as follows: a generative adversarial network model named multi-discriminator collaborative GAN (MDGAN) is proposed, which adds a multi-discriminator module after the generator for batch discrimination of real and augmented datasets. Each discriminator will evaluate the incoming data and feed back the evaluation results to the generator. The MDGAN model can change the offset effect of the generated fake data images on the two datasets by assigning different weights to the real and augmented datasets. For example, if the weight value of the real dataset is higher, the generated fake images will be closer to the real images. On the other hand, if the weight value of the augmented dataset is higher, the generated fake images will be more biased towards the augmented dataset. Therefore, in order to reduce the probability of sample penetration, the weight value of the real dataset in this paper is set to be always greater than the weight value of the augmented dataset.

[0037] The application will be further described below with reference to the accompanying drawings and examples.

[0038] Example 1:

[0039] A multi-discriminator fusion collaboration module is improved for GAN discriminators. When GAN has only a single discriminator, the augmented dataset and the original dataset will be mixed together as real dataset input to the discriminator, while the discriminator will also receive the fake data generated by the generator for judgment. Since a single discriminator cannot distinguish between the original dataset and the augmented dataset, it is easy to cause the sample penetration problem.

[0040] To solve this problem, the method in this chapter introduces a second discriminator in addition to the original discriminator to assist the original discriminator, thereby constructing a multi-discriminator module.

[0041] The training dataset of the second discriminator is different from that of the original discriminator, but the discriminator will still judge the images generated by the generator and feed back the judgment results to the generator. The objective function of the improved multi-discriminator is as follows.

[0042] maxD Final =W D maxD+W DAug maxD Aug

[0043] wherein D Final is the final discriminator return value, W D is the weight of the original discriminator output value feedback to the generator, and W D_Aug is the weight of the enhanced discriminator output value feedback to the generator, and W D and W D_Aug are equal to 1. In order to make the data distribution of the discriminator close to the real data set, W D is greater than W D_Aug , and the value in the present application is W D = 0.8, W D_Aug = 0.2. That is, the weight of the discriminator inputting the real data set feedback to the generator is 80%, and the weight of the discriminator inputting the enhanced data set feedback to the generator is 20%, which can effectively reduce the occurrence of sample penetration problems.

[0044] In a single discriminator, all image data are first subjected to two-dimensional convolutional layers for feature extraction, then input into multiple residual blocks to prevent neural network overfitting, and a dropout layer is added between every two residual blocks to reduce the overfitting ability of the discriminator. Then, the output result is subjected to a pooling layer for down-sampling, the above operation is repeated multiple times, and finally a fully connected layer is used for the output of the result.

[0045] The present application selects three public data sets as original data for experiment when training, which are OxfordFlowers102, CelebA and Cathedral three data sets. The data set used in embodiment 1 is shown in the test set in table 1:

[0046] Table 1: Test set information used in embodiment 1:

[0047] Dataset Number Size Whether public Oxford Flowers 102 8000 64*64 Public CelebA 8000 64*64 Public Cathedral 8000 64*64 Public

[0048] Tables 2-4 summarize the results of parameter optimization in embodiment 1 of the present application. Since MDGAN uses multiple discriminators to evaluate the authenticity of generated images, the generator loss function needs to be adjusted accordingly. In the improved generator loss function, W_D and W D_Aug ​Here, W_D represents the weight of the loss value returned by the discriminator corresponding to the original dataset and the augmented dataset in the total loss value, respectively, where n is the total number of discriminators. Since MDGAN uses two discriminators as the components of the multi-discriminator module, the total number of discriminators n is set to 2. To balance the impact of the original dataset and the augmented dataset on the generator, we conducted parameter optimization experiments. Because the augmented dataset introduces sample percolation problems to the generator, the weight values ​​of the augmented discriminators are always less than the weight values ​​of the original discriminators, and the weight values ​​W_D_ of the original discriminators and the weight values ​​W_D_ of the augmented discriminators are... Aug The ratio should be as small as possible. Tables 3-3 to 3-6 show the ratios of W_D_ on different datasets when W_D = 1 is fixed. Aug The impact of W_D_ on the optimal FID value during training: To avoid excessive influence of augmented datasets on the results, causing sample perpetuation, W_D_ is generally set... Aug The value should be kept below 0.5.

[0049] Table 2: Parameter optimization experiment of the present invention on the OxFlowers102 dataset when WD=1 in Example 1.

[0050] WD_Aug FID Number of iterations Training time / h 0.05 25.17 66500 22 0.1 23.66 72500 24 0.15 22.02 75500 25 0.2 20.70 58000 19 0.25 19.60 54000 18 0.3 22.15 74000 25 0.35 23.90 67000 22 0.4 24.45 65500 22 0.45 25.47 71500 24 0.5 25.16 77000 26

[0051] Table 3: Parameter optimization experiments of the present invention on the CelebA dataset when WD=1 in Example 1.

[0052] WD_Aug FID Number of iterations Training time / h 0.05 25.73 74000 25 0.1 24.49 72500 24 0.15 23.36 66000 22 0.2 22.61 57500 19 0.25 20.36 59500 20 0.3 21.52 62000 21 0.35 22.67 63500 21 0.4 23.59 69000 23 0.45 25.35 70500 23 0.5 26.14 64500 21

[0053] Table 4: Parameter optimization experiments of the present invention on the CelebA dataset when WD=1 in Example 1.

[0054] WD_Aug FID Number of iterations Training time / h 0.05 23.75 62000 21 0.1 22.16 63500 21 0.15 18.93 67000 22 0.2 16.92 59500 20 0.25 19.28 55000 18 0.3 21.57 61500 20 0.35 25.72 64500 21 0.4 25.84 71000 24 0.45 27.61 69500 23 0.5 26.95 73500 24

[0055] Table 5 summarizes the results of training different methods on multiple datasets, and Figure 6 The training process of the MDGAN and AdaFM methods on the Oxford Flowers 102 training set is demonstrated. Through observation, we can draw the following conclusions:

[0056] (1) By combining a generator with multiple discriminators, the MDGAN method significantly improves the quality of generated images. This method effectively reduces the probability of sample percolation, thereby avoiding over-learning of the data distribution of the dataset after the generated images are augmented.

[0057] (2) Because WGAN-GP and TransferGAN use larger network model architectures, this makes them unable to successfully train in the case of a small amount of data set. In the final training results, the optimal FID value reached by the MDGAN model is better than other methods on three data sets. Compared with AdaFM, the final result of MDGAN is optimized by about 29% at most and 23% at least. It can be seen that MDGAN has a greater improvement in improving the quality of the generated image.

[0058] Table 5: Optimal FID values of different methods in Example 1 on different data sets

[0059]

[0060]

[0061] Example 2

[0062] To verify the training results of the multi-discriminator fusion writing module under a small amount of data set, the MDGAN and AdaFM methods are trained under a small amount of data set. The data set used for training is Flowers-25, which only has 25 images. Finally, the training results of each method are calculated and drawn into Table 6. From Table 6, when the data set only has 25 images, the FID value of the image generated by the AdaFM method is 85.4. By contrast, thanks to the further improvement made by the method of this chapter, the final FID value of MDGAN is 78.98, which is better than the result of AdaFM. Figure 7 To verify the training results of the multi-discriminator fusion writing module under a small amount of data set, the MDGAN and AdaFM methods are trained under a small amount of data set. The data set used for training is Flowers-25, which only has 25 images. Finally, the training results of each method are calculated and drawn into Table 6. From Table 6, when the data set only has 25 images, the FID value of the image generated by the AdaFM method is 85.4. By contrast, thanks to the further improvement made by the method of this chapter, the final FID value of MDGAN is 78.98, which is better than the result of AdaFM. Figure 8 To verify the training results of the multi-discriminator fusion writing module under a small amount of data set, the MDGAN and AdaFM methods are trained under a small amount of data set. The data set used for training is Flowers-25, which only has 25 images. Finally, the training results of each method are calculated and drawn into Table 6. From Table 6, when the data set only has 25 images, the FID value of the image generated by the AdaFM method is 85.4. By contrast, thanks to the further improvement made by the method of this chapter, the final FID value of MDGAN is 78.98, which is better than the result of AdaFM.

[0063] Table 6: Training results of MDGAN and AdaFM on Flowers-25 data set

[0064] FID Epoch FID AdaFM 85.4 65000 22h MDGAN 78.98 54000 18h

Claims

1. An improved GAN model method for small dataset scenarios, characterized in that, The steps are as follows: Step 1: Input a random vector z that follows a Gaussian distribution into the generator G. The generator G generates a fake image G(z) based on the input random vector. Step 2: Input the fake image G(z) generated by generator G into the discriminator fusion module for gradient update; The discriminator fusion module has the following structure: The discriminator fusion module includes a raw discriminator and an enhanced discriminator; The training dataset for the enhanced discriminator is different from that of the original discriminator, but the discriminator will still judge the images generated by the generator and feed the judgment results back to the generator. The objective function of the improved multi-discriminator is as follows: maxD Final =In D maxD+W D_Aug maxD Aug Among them, D Final W is the final discriminator return value. D W represents the weights fed back to the generator from the original discriminator output value. D_Aug The output value of the enhanced discriminator is fed back to the generator as a weight, and W D With W D_Aug The sum of is 1; In a single discriminator, all image data is first processed through a two-dimensional convolutional layer for feature extraction, and then input into multiple residual blocks to prevent overfitting of the neural network. A dropout layer is added between every two residual blocks to reduce the discriminator's overfitting ability. Next, the output is downsampled through a pooling layer. The above operation is repeated multiple times, and finally, a fully connected layer is used to output the result. Step 3: After receiving the fake image G(z) from the generator G, the discriminator fusion module inputs the original real image Xori and the fake image G(z) into the original discriminator Dori for identification; at the same time, it inputs the enhanced real image Xaug and the fake image G(z) into the enhanced discriminator Daug for identification. Step 4: Weight the loss values ​​output by the original discriminator Dori and the enhanced discriminator Daug; Step 5: Feed the mixed result value back to the generator G for gradient update; The objective function of the improved discriminator remains unchanged compared to the original discriminator; the only difference is that the discriminator is divided into two identical components for computation, which are then integrated at the end, as shown below: The binary cross-entropy loss function is chosen as the loss function in the model, and its original calculation formula is shown below: Since the loss function used is binary cross-entropy, the loss function of the improved generator can be expressed as:

2. The method for improving GAN models for small dataset scenarios as described in claim 1, characterized in that, The W mentioned D =0.8, W D_Aug =0.2.

Images