A two-stage image generation method based on a generative adversarial network

Abstract

The application relates to a two-stage image generation method based on a generative adversarial network, which comprises the following steps: an image feature generation step: a residual network is used to capture the features of an input image, a feature generation network is trained, a feature generator is obtained by using a GAN to fit the features of the image after the image passes through the capturing network, and a discriminator is used to distinguish the features of the image after the image passes through the capturing network and the features generated by the feature generator; and an image generation step: the parameters of the feature generator trained in the image feature generation step are fixed and added to a new image generation network, random noise is allowed to pass through the feature generator first, and then the output is transmitted into an image generator to obtain a finally generated image. In the first stage, a generation network is used to learn the features of the image, and then in the second stage, an adversarial network is used to generate the image. Compared with a traditional GAN optimization method, the training difficulty of the GAN can be reduced, the diversity of the generated mode can be improved, and the image generation quality can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and particularly relates to a two-stage image generation method based on a generative adversarial network. BACKGROUND

[0002] Deep learning has a very bright performance in the field of computer vision, and methods based on deep learning can process many image-related problems, such as image generation and image repair. As an important method in deep learning, the generative adversarial network (GAN) can generate high-quality images for data set expansion, image classification, etc. The generative adversarial network can effectively learn the real distribution, and can complete the generation task through a generator and a discriminator. GAN can imitate real images to generate images of the same type, which can solve the problem of insufficient training data in some experiments, and GAN also solves the problem of insufficient labeled data when generating images. GAN can also convert dialogues and texts into images. Therefore, the research on GAN has great significance and can solve problems in various fields.

[0003] Although the generative adversarial network performs well, it has disadvantages and is difficult to train. GAN is a model in which the generator and the discriminator are constantly updated and optimized until the Nash equilibrium state. Although GAN has a theoretical optimal solution, in actual operation, GAN has many problems, such as unstable training, gradient disappearance, and mode collapse. The optimization goal of the original GAN is the cross-entropy between real data and generated data, which reduces the cross-entropy to make the difference between the generated data and the real data distribution smaller, and the generated data more like the real data. If the discriminator is optimal during training, the loss of the generation model becomes a fixed constant, which causes the gradient of the generation model to disappear during training, and the entire network cannot be further updated and optimized. If the discriminator is not well trained, the discriminator also gives "encouragement" to bad generated samples, so the generator will continue to generate similar bad samples, resulting in poor sample generation effect, and the discriminator and the generator do not know each other. Only when the discriminator is trained just right, its discrimination ability is just right, at this time the entire network can be constantly optimized in the correct direction to generate samples close to the real samples. Therefore, one of the reasons why GAN is difficult to train is that it is difficult to grasp the actual training.

[0004] Because GAN has the problem of unstable training, sometimes the quality of the generated images is uneven and lacks diversity, and often needs to be optimized. However, the model training of these works is essentially the idea of the original GAN, so there are still problems of unstable training, insufficient diversity of generated samples, and poor image generation quality. SUMMARY

[0005] The present application aims to overcome the shortcomings of the prior art, and provides a two-stage image generation method based on a generative adversarial network, which solves the problem that the prior art does not optimize the GAN from the training process of the generative adversarial network, resulting in unstable GAN training, insufficient sample diversification, and poor quality of generated images.

[0006] The purpose of the present application is achieved by the following technical scheme: a two-stage image generation method based on a generative adversarial network, the two-stage image generation method comprising:

[0007] An image feature generation step: capturing the features of an input image through a residual network, training a feature generation network, using GAN to fit the features of the image after the capturing network to obtain a feature generator, and distinguishing the features of the image after the capturing network and the features generated by the feature generator through a discriminator;

[0008] An image generation step: fixing the parameters of the feature generator trained in the image feature generation step, adding them to a new image generation network, letting random noise pass through the feature generator first, and then outputting it into the image generator to obtain the finally generated image, and the discriminator distinguishes the features of the real image and the image generated by the image generator, and guides the generation of the feature generator.

[0009] The discriminator distinguishes the features of the real image and the image generated by the image generator, and guides the generation of the feature generator, which includes: the discriminator distinguishes the features of the real image and the image generated by the image generator, and if the features of the real image and the image generated by the image generator do not meet the requirements, the features at this time are used as the learning object of the feature generator to regenerate the feature generator until the features of the real image and the image generated by the image generator meet the requirements.

[0010] In the image feature generation step, the image is subjected to convolution operation through a residual network to generate a feature map, which contains the feature information of the image, and the residual network continuously operates to output the feature map as classification information.

[0011] The loss function used in the image feature generation step and the image generation step is: Where GP is the gradient penalty term, CT is , represents the continuity regularization term, G=G i (G f ), G i represents the image generator, G f represents the feature generator, i.e. random noise first passes through G f , and then its output passes through G i .

[0012] The present application has the following advantages: a two-stage image generation method based on a generative adversarial network uses a two-stage model, which learns the features of an image in the first stage through a generation network, and then generates an image in the second stage through an adversarial network. Compared with the traditional GAN optimization method, the training difficulty of GAN can be reduced, the diversity of the generated mode can be improved, and the image generation quality can be improved. BRIEF DESCRIPTION OF DRAWINGS

[0013] Figure 1 The flowchart of the present application is shown in the figure;

[0014] Figure 2 The structure diagram of the feature generation model is shown in the figure;

[0015] Figure 3 The structure diagram of the image generation model is shown in the figure;

[0016] Figure 4 The comparison diagram of MNIST experimental results is shown in the figure;

[0017] Figure 5 The comparison diagram of Fashion MNIST experimental results is shown in the figure;

[0018] Figure 6 The comparison diagram of CIFAR-10 experimental results is shown in the figure;

[0019] Figure 7 The IS score change diagram in the CIFAR-10 experiment is shown in the figure. DETAILED DESCRIPTION

[0020] The present application will be further described below in conjunction with the accompanying drawings, but the protection scope of the present application is not limited to the following description.

[0021] The present application proposes a two-stage training image generation method of a generative adversarial network. Compared with the single complete training process of the traditional GAN training, the present application divides the GAN training into two stages to reduce the training difficulty and stabilize the training process, thereby generating higher quality images. In the first stage, a classification network is first trained to extract the features of the input image, and then the trained classification network is frozen to its parameters, and a generative adversarial network is used to fit the features of the image after the classification network; in the second stage, the parameters of the feature generator in the first stage are frozen and added to a new generative adversarial network, so that the input random noise is first converted into an image through the feature generator and then through the image generator, and then the discriminator distinguishes between the generated and real images. Through the two-stage training, the training difficulty of the single complete training process is reduced, the stability of the training is improved, and the image generation quality is improved; specifically including the following contents:

[0022] As Figure 1As shown, first input an image data to perform the first stage training; in the first stage, the residual network is used to capture the features of the image. Then a feature generation network is trained, and the GAN is used to fit the features of the image after the feature capture network, so as to obtain a feature generator. The discriminator distinguishes the features of the image after the capture network and the features generated by the feature generator. In the second stage, i.e., the image generation stage. The feature generator obtained in the first stage is fixed, and is added to the new image generation network. Let the random noise pass through the feature generator first, and then output into the image generator to obtain the finally generated image. The discriminator distinguishes the real image and the image generated by the image generator, and guides the generation of the generator. If the captured image features in the first stage can optimize the mapping T of the generator, and map the original Gaussian distribution to the feature distribution, it is helpful for subsequent image generation and improves the quality of image generation.

[0023] In the first stage of model training, a feature generation model needs to be trained, which is also essentially a generative adversarial model. The learning object of the generator changes from an image to a feature. As shown in Figure 2 , the image is transmitted to the feature capture network, and then the feature is output. The generator learns the feature, and the discriminator discriminates the generated feature of the generator and the real feature. The two constantly play and evolve. The feature capture network is an image classification network. The image passes through the convolution operation of the classification network to generate a feature map, which contains the information of the image. The network continuously operates to output the feature map as classification information. The information in the network feature map can contain the feature information of the image and distinguish the image. Therefore, if the feature is learned and used in the generative adversarial network, the generation ability of the model can be improved. The classification network based on the residual network ResNet is adopted as the feature capture network in the present application. Figure 2 The feature generator in the is essentially a generative adversarial network. Unlike common image generation, the learning object of the feature generator changes to a feature. In the original GAN, random noise passes through the generator to generate an image, and the discriminator distinguishes the real and generated images. As shown in Figure 2 , in the feature generation network, the input image passes through the feature capture network to output the feature; the random noise z passes through the feature generator to generate data similar to the feature; and the discriminator distinguishes the real feature and the generated feature.

[0024] In the second stage of model training, the image generator and the discriminator of the CT-GAN structure are adopted, as shown in Figure 3 . Unlike the original CT-GAN, the feature generator G f is added before the generator. The random noise z passes through the feature generator G fThen the features are input into the image generator, and the discriminator discriminates the real image and the generated image. The residual block, down-sampling, Dropout structure and method are used in the discriminator of the image generation stage; while the residual block, up-sampling and other structures and methods are used in the generator.

[0025] Since the feature generation model of the first stage and the image generation model of the second stage are both GAN models, similar methods can be used in network structure and training method. The feature generation model and the image generation model use the same loss function as follows:

[0026]

[0027] Where GP is the gradient penalty term in WGAN-GP, CT is the continuity regularization term newly added in CT-GAN, and the specific expression of CT is as follows:

[0028]

[0029] Since the generator is not completely the structure of CT-GAN, G in the objective function can be regarded as the nested function of the image generator and the feature generator, that is,

[0030] G=G i (G f )

[0031] Where G i represents the image generator, and G f represents the feature generator, that is, the random noise first passes through G f , and then the output thereof passes through G i .

[0032] When generating images in the second stage, the network first trains the generator. For random noise z, it is input into the feature generator G f , to obtain the feature G f (z), and then the image is generated. The discriminator discriminates the generated image and the real image as the classical GAN network.

[0033] Experiment 1: MNIST dataset experiment

[0034] As Figure 4As shown in the figure, the left side is the picture generated by the corresponding base model, and the right side is the picture generated by the two-stage model of the application. It can be seen from the vision that the right picture is more detailed than the left picture, and more like a handwritten number. The left number is more distorted and blurred. From the quantitative indicators, the FID score of the base model is 8.14, and the FID score of the two-stage model of the application is 5.27. According to the FID measurement standard, the lower the FID score, the better the picture quality, which reflects that the generation effect of the two-stage model is better than that of the base model on the MNIST data set, verifying the effectiveness of the model.

[0035] Experiment two: Fashion MNIST experiment

[0036] As Figure 5 shown, the right side is the picture generated by the two-stage model, and the left side is the picture generated by the corresponding base model. It can be seen from the vision that the right picture is more real and vivid than the left picture. The blur and collapse picture angle of the left picture, such as the first pattern in the upper left corner, is difficult to identify its shape and specific category. In addition, the FID score of the base model is 11.94, and the FID score of the two-stage model of the application is 8.25, reflecting that the generation effect of the two-stage model is better than that of the base model on the Fashion MNIST data set.

[0037] Experiment three: CIFAR-10 experiment

[0038] First, from the comparison of the generated images, as Figure 6 shown, the left picture is the picture generated by the base model of the present application, and the right picture is the picture generated by the two-stage model. It can be seen from the vision that the right picture is more vivid and more real, such as the "horse" in the right picture is more lively than the left picture. From the quantitative indicators IS and FID, as shown in Table 1, the IS score of the two-stage model of the application is significantly higher than the IS score of the base model, and the FID score is lower than the score of the base model, which shows that the two-stage generation method proposed in the application is effective in generating images and can improve the image quality of GAN. Compared with other GAN models, it can be seen that the model of the present application has strong performance and can be better than most models.

[0039] Table 1 Comparison of IS and FID results of different models on CIFAR-10 data set

[0040] Methods Inception score FID WGAN-GP 7.86±0.07 28.2±0.33 CT-GAN 8.12±0.13 22.4±0.26 SN-GAN 8.22±0.05 21.7±0.21 BWGAN 8.26±0.07 - MGAN 8.33±0.10 26.74 Ours(ResNet-56) 8.30±0.09 16.51 Ours(ResNet-110) 8.37±0.08 15.85

[0041] Figure 7The specific changes of IS scores during the training process of the two-stage model are shown. Compared with the base model and the two-stage generation model, the IS score of the initial base model increases faster, but it is "weak" in the later stage, and the two-stage generation model can have a higher score in the later stage. Comparing the two models in the two-stage generation model using different feature capture networks, it can be seen that the model using ResNet-110 is higher than the model using ResNet-56 in the IS during the training process, which shows that using a feature capture network with higher classification accuracy in the first stage can improve the final generation quality of the two-stage model.

[0042] The above merely describes the preferred embodiments of the present application, and it should be understood that the present application is not limited to the forms disclosed herein, and should not be considered as excluding other embodiments, but can be used in various other combinations, modifications and environments, and can be modified within the scope of the concepts described herein by the above teachings or related art or knowledge. Any modification and change made by those skilled in the art without departing from the spirit and scope of the present application shall be within the protection scope of the claims of the present application.

Claims

1. A two-stage image generation method based on generative adversarial networks, characterized in that: The two-stage image generation method includes: Image feature generation steps: Capture the features of the input image through a residual network, train a feature generation network, use GAN to fit the features of the image after passing through the capture network to obtain the feature generator, and use a discriminator to distinguish the features of the image after passing through the capture network from the features generated by the feature generator; Image generation steps: The parameters of the feature generator obtained from the image feature generation step are fixed and added to the new image generation network. Random noise is first passed through the feature generator, and then the output is fed into the image generator to obtain the final generated image. The discriminator distinguishes the feature differences between the real image and the image generated by the image generator and guides the generation of the feature generator. The discriminator distinguishes the feature differences between the real image and the image generated by the image generator, and guides the generation of the feature generator by: the discriminator distinguishes the feature differences between the real image and the image generated by the image generator; if the feature differences between the real image and the image generated by the image generator do not meet the requirements, the features at this time are used as the learning object of the feature generator to regenerate the feature generator until the feature differences between the real image and the image generated by the image generator meet the requirements. The loss function used in both the image feature generation step and the image generation step is: Where GP is the gradient penalty term, and CT is... Represents a continuity regularization term. , G i Represents an image generator. G f This represents a feature generator, where random noise first passes through... G f Then its output passes through G i .

2. The two-stage image generation method based on generative adversarial networks according to claim 1, characterized in that: In the image feature generation step, the image is processed by a residual network through convolution operations to generate a feature map. The feature map contains the feature information of the image. The residual network then outputs the feature map as classification information through continuous operations.

Images