An image sample expansion method based on a generative adversarial network

Abstract

The application discloses an image sample expansion method based on a generative adversarial network, for a single frame image, the internal distribution in each small block in a given image is captured, the generation of the image sample starts from the lowest resolution scale, sequentially passes through all the generators until the highest resolution scale, and meanwhile, noise is added in each scale; after the image is generated, the consistency loss idea of the cycle generative adversarial network CycleGAN is introduced, the generated image is converted back to the original image through the generator to determine the mapping relationship between the original image and the generated image. The application establishes a stable and firm mapping relationship by introducing the cycle generative adversarial design in the CycleGAN network, and ensures the stability of the generated image.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of computer image processing, and particularly relates to an image sample expansion method based on a generative adversarial network. BACKGROUND

[0002] In the field of computer vision, a deep network model needs a large amount of training data, when the data set is small, the network trained is easy to overfit, and the model will not be robust. When processing general natural images containing complex structures and textures, the generated images often appear distorted or the style is completely changed, resulting in completely different distribution structures from the original data.

[0003] In some fields where image samples are scarce, the size of the sample set is expanded by using a few-sample generation technology, so that the deeper and more complex network trained can resist complex and variable actual scenes and has stronger generalization ability. SUMMARY

[0004] In view of the situation that image samples are scarce in some fields, an image sample expansion method based on a generative adversarial network is proposed.

[0005] The technical scheme adopted by the application to solve the technical problem is: an image sample expansion method based on a generative adversarial network, which captures the internal distribution in each small block in a given single frame image through a SinGAN model, and sequentially passes the image sample through all the generators from the lowest resolution scale until the highest resolution scale, while adding noise in each scale, all the generators and discriminators have the same receptive field, and the structure of size reduction can be captured in the generation process, and new images with high quality and diversity are generated with the same visual content as the given image; after the new image is generated, the consistency loss idea of the cycle generative adversarial network CycleGAN is introduced, the generated image is converted back to the original image through the generator for discriminant determination, the mapping relationship between the original image and the generated new image is constrained, and a Cycle-SinGAN network integrating the characteristics of SinGAN and CycleGAN is obtained; the SinGAN model adopts a single image patch, each SinGAN model is responsible for capturing the distribution of patches of different scales, and a pyramid composed of multiple fully convolutional GANs {G0, G1, …, G N} is obtained; each generator G N generates the weight and bias of the real image x N , the image sample starts from the coarsest scale, adds noise Z N , sequentially passes through all the generators G N , obtains the image x N , and after the image x N is enlarged by r times, another noise Z N－1input generator G N－1 , generate more detailed image x N－1 , and so on, until the finest scale, generate image x0 with rich details: Where the pyramid scale factor formula is

[0006] The image sample expansion method based on the generated adversarial network, the CycleGAN generates images through a consistency loss function: L cyc (G, F) = E x～pdata(x) [|| F(G(x)) - x ||1] + E y～pdata(y) [|| G(F(y)) - y ||1], wherein G(x) represents the change from the original image domain to the generated image domain, F(G(x)) represents the change from the generated image domain to the original image domain, the model training hopes that F(G(x)) is as close to x as possible, and the L1 distance constraint is adopted, and similarly, G(F(y)) is as close to y as possible, and the consistency loss ensures the stability of the mapping relationship between the original domain and the generated domain.

[0007] The beneficial effects of the present application are: the network structure of Cycle-SinGAN proposed by the present application combines the network characteristics of SinGAN and CycleGAN, SinGAN generates new images with high quality and diversity by capturing the internal distribution in each small block of the given image, and the visual content of the generated image is the same as that of the given image. The present application introduces the cycle generation adversarial design in the CycleGAN network to establish a stable and firm mapping relationship, and ensures the stability of the generated image. BRIEF DESCRIPTION OF DRAWINGS

[0008] Figure 1 is the basic structure diagram of the SinGAN network of the present application;

[0009] Figure 2 is the traditional CycleGAN network structure diagram;

[0010] Figure 3 is the improved CycleGAN network structure diagram by introducing the Inception module;

[0011] Figure 4 is the structure diagram of the Cycle-SinGAN network of the present application;

[0012] Figure 5 is the generation effect diagram of different scene and different scale images. DETAILED DESCRIPTION

[0013] The present application will be further described in detail below in combination with the drawings.

[0014] This paper first proposes a Cycle-SinGAN network structure that combines the characteristics of SinGAN and CycleGAN networks. The SinGAN model can capture the intrinsic distribution of each small patch in a given image and then generate new images with the same high-quality and diverse visual content as the given image.

[0015] SinGAN is similar to traditional GAN ​​settings. The training samples are not the entire image samples in the database, but patches of a single image. The training images are not limited to specific images, but general natural images with complex structures and textures are used. This selection can obtain internal information of the image at more scales. This is achieved through a pyramid composed of fully convolutional lightweight GANs, each of which is responsible for capturing the distribution of patches of different scales. The structure of the SinGAN model is as follows: Figure 1 As shown, there are multiple fully convolutional GANs {G0, G1, ..., G N}, these fully convolutional GANs are responsible for learning the image {x0, x1, ..., x N}, different GANs learn different sizes of patches, gradually generating new images from coarse to fine, from low resolution to high resolution. These new images can have high variability while retaining the global structure and detailed texture of the given training image. In these fully convolutional GANs, each generator G N will generate a real image x N The image sample starts from the coarsest scale and adds noise Z N , through all generators G in turn N , get the image x N , the image x N After amplifying r times, add another noise Z N－1 , and input into the generator G N－1 , generate a more detailed image x N－1 , and so on, until the finest scale, generating an image x0 with rich details. Its basic formula is and

[0016] Image samples are generated starting from the lowest resolution scale and progressing through all generators to the highest resolution scale, while adding noise at each scale. All generators and discriminators have the same receptive field, so they can capture structures of decreasing size during generation. At the lowest resolution scale, the effective receptive field typically covers a large fraction of the image, allowing the image's structural layout to be captured. Each generator Gn has a finer scale, adding details not present at the previous scale.

[0017] The basic principle of SinGAN super-resolution reconstruction is to train the model on a low-resolution image, set the reconstruction loss weight, and the pyramid scale factor formula is Since small structures tend to repeat in natural images, the image is upsampled r times, and it is injected into the last generator G0 along with noise to learn to generate image samples, and this process is repeated k times to obtain the final high-resolution output. The discriminator D N are all based on patch judgment, from D N to the D0 layer.

[0018] After image generation, the mapping relationship established between the generated image and the original image needs stronger constraints to ensure that the generated image has a similar distribution form as the original image. Therefore, on the basis of the SinGAN network, the consistency loss idea of the CycleGAN is introduced, and the generated image is converted back to the original image through the generator to constrain the mapping relationship between the original image and the generated image. The traditional Cycle-GAN network generator adopts a residual module, which is connected by full convolution, composed of an encoder, a converter and a decoder, and its structure is as shown in Figure 2 . But the residual module often has better performance in deep network structure, so the Inception module is introduced in the traditional CycleGAN network to replace the original single residual module, and the improved network structure is as shown in Figure 3 .

[0019] The most important part of the improved CycleGAN network acting on the generated image is its consistency loss function, and its formula is as follows: L cyc (G,F)=E x～pdata(x) [||F(G(x))-x||1]+E y～pdata(y) [||G(F(y))-y||1], where G(x) represents the change from the original image domain to the generated image domain, F(G(x)) represents the change from the generated image domain to the original image domain, and the model training hopes that F(G(x)) is as close to x as possible, and the L1 distance constraint is adopted. Similarly, G(F(y)) is also as close to y as possible. This consistency loss ensures the stability of the mapping relationship between the original domain and the generated domain. Therefore, the network structure of the final Cycle-SinGAN is as shown in Figure 4 .

[0020] The Cycle-SinGAN network is summarized as follows: for a single frame image, the internal distribution in each small block in the given image is captured, the generation of the image sample starts from the lowest resolution scale, and then passes through all the generators in turn until the highest resolution scale, and noise is added at each scale. All the generators and discriminators have the same receptive field, which can capture the structure of size reduction in the generation process, and then generate new images with high quality and diversity with the same visual content as the given image. After image generation, the consistency loss idea of the cycle generative adversarial network CycleGAN is introduced, and the generated image is converted back to the original image through the generator for discriminant judgment, to constrain the mapping relationship between the original image and the generated image.

[0021] Since the network of Cycle-SinGAN is mainly used to capture the internal distribution of each small block in a single frame image, the image dataset adopts natural data and data with rich texture details, which can achieve better results. Therefore, the present application adopts ImageNet val, natural image data crawled on the Internet and images taken by a mobile phone to constitute a training data set, and divides the training test set according to a ratio of 9:1. Figure 5 As shown in the figure, the generation effect of the trained model on different scene and different scale images can be seen. The network can automatically repair some blurred details and textures in the original image, and has a certain reconstruction enhancement effect on the image. For some small sample fields, such as coal mining and medical assistance, the network has a good data enhancement effect.

[0022] The following table is the NIQE score of the present application and each method on each data set:

[0023] GAN WGAN BigGAN ESRGAN Cycle-SinGAN RealSR 6.1692 6.1575 6.2440 6.1242 4.4312 DRealSR 6.8758 8.4551 7.7545 7.4722 4.9156 DPED 6.4565 5.4648 6.2125 5.4156 5.1326 OST300 4.4440 3.5245 5.0232 4.5484 2.8192 ImageNet val 7.4334 3.2547 6.1555 3.4648 4.7438 ADE20K val 7.4845 3.4854 6.4545 3.4102 3.9555

[0024] As can be seen from the above table, the Cycle-SinGAN algorithm adopted by the present application has better image generation effect on RealSR, DRealSR, DPED, OST300 and other data sets compared with the original GAN, WGAN, BigGAN, ESRGAN and other methods. The present application tests and compares the natural image quality evaluator (NIQE) index on RealSR-Camon, RealSR-Nikon, DRealSR, DPED-iphone, OST300, ImageNet val, ADE20K val and other general verification sets by using the original GAN, WGAN, BigGAN, ESRGAN and other methods, which verifies the advancement of the present method.

[0025] The data set and platform environment of the experiment in this paper are as follows:

[0026] The training data set is composed of ImageNet val, natural image data crawled on the Internet and images taken by a mobile phone. The test data set is divided from the training data set in a ratio of 9:1, and the total number of images is 1000. The hardware and software platforms include: Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz * 12; GPU: Force GTX1080Ti; operating system: ubuntu 16.04 LTS; deep learning framework: pytorch. The network is trained. The initial learning rate is set to 0.003, the decay coefficient is set to 0.0005, the cosine annealing learning rate adjustment strategy is adopted, and the iteration epoch is set to 200.

[0027] The above examples only exemplarily illustrate the principles and effects of the present application and part of the applied examples. For those skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are all within the protection scope of the present application.

Claims

1. A method for image sample expansion based on a generative adversarial network, characterized by: The SinGAN model captures the intrinsic distribution of each small block in a given single-frame image, passes the image samples through all generators in sequence, starting from the lowest resolution scale to the highest resolution scale, and adds noise at each scale to generate a new image with the same visual content as the given image; The new image is introduced into the Cycle Generative Adversarial Network (CycleGAN), and the generated image is converted back through the generator to be identified and judged with the original image, constraining the mapping relationship between the original image and the new image, and obtaining the Cycle-SinGAN network that combines SinGAN and CycleGAN; the SinGAN model uses a single image patch, and each SinGAN model captures the patch distribution of different sizes, resulting in a Cycle-SinGAN network composed of multiple fully convolutional GANs {G0, G1, ..., G N }composed of a pyramid; each generator G N Generate real image x N The weights and biases of the image samples start from the coarsest scale and add noise Z N , through all generators G in turn N , get the image x N , the image x N After amplifying r times, add another noise Z N－1 , and input into the generator G N－1 , generate a more detailed image x N－1 , and so on, until the finest scale, generating an image x0 with rich details: The pyramid scale factor formula is 2. The image sample expansion method based on generative adversarial network according to claim 1, characterized in that: The CycleGAN generates images through the following consistency loss function: L cyc (G,F)=E x～pdata(x) [||F(G(x))-x||1]+E y～pdata(y) [||G(F(y))-y||1], Where G(x) represents the change from the original image domain to the generated image domain, F(G(x)) represents the change from the generated image domain to the original image domain, F(G(x)) is as close to x as possible, and G(F(y)) is as close to y as possible.

Images