A method, device and storage medium for alleviating mode collapse in GANs training

Abstract

The application discloses a GANs training method and device for relieving mode collapse and a storage medium, and relates to the technical field of generative adversarial network mode collapse. The training method of cutting and then distributing fitting of original images is used to relieve the problems caused by mode collapse. Based on the framework of GAN image generation, the original images are cut in the stage of processing the original images, different-size original image patches are generated, the global features of the images are better learned, and the diversity of the generated images is improved. In the process of generating the images, the constraint parameters of the image patches and feature map patches are used in the latent space, and the penalty term is correspondingly increased. In view of the repetition of the image feature information contained in different convolution layers, the weight of the specified convolution layer is reduced to avoid repeated extraction of the feature information. The application can make the generator generate more diverse images and include more global features of the images, is favorable for relieving the problem of mode collapse, and thus better GANs training results are achieved.

Description

Technical Field

[0001] This invention relates to the field of pattern collapse technology for Generative Adversarial Networks (GANs), and in particular to a GANs training method, device, and storage medium for mitigating pattern collapse. Background Technology

[0002] Image synthesis plays an increasingly important role in many multimedia applications, leading to growing interest in research related to image synthesis, such as GANs. However, the pattern collapse problem is one of the most significant challenges in GAN applications. The GAN framework involves simultaneously training two sub-networks: a generator to produce samples and a discriminator to distinguish between real and fake samples. Ideally, the generator captures the real data distribution and generates similar new samples. However, the GAN framework primarily focuses on generating visually consistent images, which can lead to insufficient diversity in generated data when there are many image patterns. Some data patterns may be lost during the generator's learning of the original data distribution; this is the common pattern collapse problem.

[0003] To address the pattern collapse problem, researchers have proposed various approaches, including applying multiple generators, improving training strategies, adding penalty terms, using regularization, and adding distribution constraints. Existing methods that add distribution constraints focus on establishing a data generation method rather than paying more attention to the image data itself. Therefore, it is necessary to propose a pattern collapse solution based on the image data itself. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a GANs training method, device, and storage medium for mitigating mode collapse. It utilizes the pixels of the image data itself as data points and employs a cropping method to convert image samples into feature map patches through a pre-trained model. Different feature patches are used to achieve further distribution fitting, thereby better learning the distribution of the original image and obtaining more global features of the original image, thus mitigating the problems caused by mode collapse.

[0005] On the one hand, the present invention provides a GAN training method for mitigating pattern collapse, which includes the following steps:

[0006] Step 1: Sample the data images of the real dataset, and then use the generator of the Generative Adversarial Network (GAN) to randomly simulate and generate negative samples to generate the original training positive and negative sample pairs.

[0007] Step 2: Crop the original training positive and negative sample pairs to generate multiple sets of original image patches of different sizes.

[0008] Step 3: Use a pre-trained feature extraction network to transform the original image patch into a feature map patch, that is, use the pre-trained feature extraction network to extract image features from the original image patch to obtain the corresponding feature map patch;

[0009] Step 4, set the objective function of GANs during training to min G [max D V(D,G)+λ p L P (G)+λ f L f (G)], the generator and discriminator of GANs are trained and optimized alternately based on the objective function. The training stops when the preset convergence condition is met, such as when the number of training times reaches the preset upper limit or the objective function value converges, and the trained target image generator is obtained.

[0010] Where G represents the generator, D represents the discriminator, and λ p , λ f The penalty term L for the patch sample P (G) and the penalty term L for the patch sample of the specified network layer. f (G) weight coefficients, V(D,G) is the basic objective function of the generative adversarial network, which is used to make the generator generate as realistic image data as possible, and to make the discriminator more accurate in distinguishing whether the input image comes from reality or is generated by the generator.

[0011] Penalty item for patch sample This represents the penalty term of the k-th convolutional layer of the generator;

[0012] Penalty item for patch samples of specified network layers Where m represents the layer number of a specified convolutional layer in the feature extraction network. This indicates the penalty term for convolutional layer m in the specified set of convolutional layers;

[0013] Wherein, for any k-th convolutional layer, its corresponding penalty term The expression is:

[0014]

[0015] Where μ and σ represent the average value and standard deviation of the output feature map of the convolutional layer, respectively. The superscript i is the identifier of the image patch, and the subscripts g and r are used to distinguish between the generated image (output by the generator) and the real image, respectively. n represents the number of image patches.

[0016] Furthermore, when cropping the original training positive and negative sample pairs, they are sliced ​​into squares of multiple sizes with different side lengths.

[0017] Furthermore, the feature extraction network is the VGG series network, such as VGG-16 and VGG-19.

[0018] On the other hand, the present invention also provides a computer device including a processor and a memory, wherein the memory stores at least one computer program, which is loaded and executed by the processor to implement a GANs training method for mitigating pattern collapse provided by the present invention.

[0019] On the other hand, the present invention also provides a computer-readable storage medium storing at least one computer program, which is loaded and executed by a processor to implement a GANs training method for mitigating pattern collapse provided by the present invention.

[0020] The technical solution provided by this invention brings at least the following beneficial effects:

[0021] This invention alleviates the problem caused by pattern collapse by using a training method that crops the original image and then performs distribution fitting.

[0022] The GAN-based image generation framework crops the original image during the original image processing stage, generating original image patches of different sizes to better learn the global features of the image and improve the diversity of the generated images.

[0023] During the image generation process, constraint parameters for image patches and feature map patches are used in the latent space, and the penalty term is increased accordingly.

[0024] To avoid the repetitive extraction of image feature information contained in different convolutional layers, this invention reduces the weight of a specified convolutional layer.

[0025] This invention allows the generator to produce more diverse images while encompassing more global features of the image, which helps to alleviate the problem of pattern collapse and thus achieve better GANs training results. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 This is a schematic diagram illustrating a GANs training method for mitigating pattern collapse, provided in an embodiment of the present invention. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be described in detail and completely below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Generally, the components of the embodiments of the present invention described and shown in the accompanying drawings can be arranged and designed using different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of the present invention.

[0029] This invention provides a GANs training method to mitigate pattern collapse, which is a novel method for fitting generated data and real data distribution to alleviate pattern collapse.

[0030] Pattern collapse is one of the most significant problems encountered in the current applications of GANs. The GAN framework simultaneously trains two sub-networks: a generator to produce samples and a discriminator to distinguish between real and fake samples. In the best-case scenario, the generator captures the real data distribution and generates similar new samples. However, the GAN framework primarily focuses on generating visually consistent images, which can lead to insufficient diversity in generated data when there are many image patterns. Some data patterns may be lost during the generator's learning of the original data distribution; this is the common pattern collapse problem.

[0031] To address the pattern collapse problem, existing solutions primarily focus on developing new generation methods and optimizing training strategies. However, in the field of image processing, a greater focus is needed on the image itself. Therefore, this invention provides a GAN training method to mitigate pattern collapse. See [link to relevant documentation]. Figure 1 Specifically, it includes the following steps:

[0032] Step 1: Sample the data images of the real dataset, and at the same time use a generator to randomly simulate and generate negative samples to generate the original training positive and negative sample pairs;

[0033] Step 2: Crop the original training positive and negative sample pairs to obtain multiple sets of original image patches for each pair. Taking an 8×8 image as an example, slice it into squares with side lengths of 1, 2, 3, and 4, resulting in a total of 64 + 16 + 4 + 4 = 88 original image patches.

[0034] Step 3: Transform the original image patch into a feature map patch using a pre-trained model (pre-trained feature extraction network), and perform multiple iterations of training based on different feature maps from different convolutional layers. This invention does not limit the specific pre-trained model, as the optimal pre-trained model varies depending on the scenario. In this embodiment, VGG-19 is used as an example to explain the specific implementation process of this invention.

[0035] Step 4: Add corresponding penalty items for patch samples Where μ and σ represent the mean and standard deviation, respectively; the superscript i indicates the i-th patch of the feature map; the subscripts g and r indicate that the feature map is obtained from the generated image and the real image through a pre-trained model, respectively; k indicates that the feature map is the output model of the k-th convolutional layer of the training model; and n represents the total number of patches. Because the feature map... Figure 1 It is generally regarded as a multi-channel image, so its mean and standard deviation are calculated by the patch.

[0036] Step 5: Add a penalty term to the corresponding N-layer patch sample. In this embodiment, convolutional layers 1, 3, 5, 9, and 13 of VGG-19 are used to map the output feature constraints. This is because deeper layers contain more global feature information, but this information is also included in larger areas of shallower information. To avoid information duplication and balance local and global features, this embodiment assigns a smaller weight to deeper features. Specifically, it adds a penalty term for corresponding patch samples to several designated layers of the pre-trained model.

[0037] Step 6: Determine the training target min of GANs G [max D V(D,G)+λ p L P (G)+λ f L f (G)] is used for training. The generator and discriminator are trained alternately to optimize the generator. Here, G represents the generator, D represents the discriminator, and λ represents the discriminator. p , λ f The penalty term L for the patch sample P (G) and the penalty term L for the patch sample of the specified network layer. f (G) Weight coefficients, whose values ​​are empirically preset values. V(D,G) is the objective function of the generative adversarial network, which aims to make the generator produce as realistic image data as possible and to make the discriminator more accurate in distinguishing whether the input image comes from reality or is generated by the generator. That is, the embodiment of the present invention adds two terms to the objective function of the generative adversarial network: L P (G) and L f (G), where L P (G) is a penalty item. The sum of the sums, i.e.

[0038] in,

[0039] In the formula, E[·] represents the expectation, x represents the real data, i.e., the real image, G(·) represents the generator output, D(·) represents the discriminator output, z represents the input random noise, and p z (z) represents the distribution of the random noise vector, and Pdata(x) represents the distribution of the real image x. This formula means that the discriminator attempts to maximize its ability to accurately distinguish between real and generated images, i.e., to maximize the above function, while the generator attempts to minimize the above function.

[0040] This invention's method alleviates the problem of pattern collapse by cropping the original image and then performing distribution fitting training. For image generation, when there are many data patterns, the generator often fails to correctly learn the data distribution, leading to the neglect of some patterns and causing pattern collapse. Therefore, better learning of image distribution and increasing the diversity of generated images is crucial to solving the pattern collapse problem. This invention's method, based on the GAN image generation framework, crops the original image during the original image processing stage, generating original image patches of different sizes. This allows for better learning of the global features of the image, significantly improving the diversity of generated images. During image generation, this invention proposes constraint parameters for using image patches and feature map patches in the latent space and increases the penalty term accordingly. In the pre-trained model processing, the original image patches are first converted into feature map patches, and a penalty term is added for this action. Furthermore, this invention considers the repetition of image feature information contained in different convolutional layers, reducing the weight of a specified convolutional layer to avoid repeated extraction of feature information. After processing through multiple convolutional layers, a pre-set training target is obtained. This allows the generator to produce more diverse images while encompassing more global features of the image, which helps alleviate the pattern collapse problem and thus achieves better GAN training results.

[0041] In an exemplary embodiment, the present invention also provides a computer device including a processor and a memory, wherein the memory stores at least one computer program. The at least one computer program is loaded and executed by one or more processors to implement the GANs training method for mitigating pattern collapse provided in the embodiments of the present invention. In an exemplary embodiment, the present invention also provides a computer-readable storage medium storing at least one computer program, which is loaded and executed by the processor of the computer device to implement the GANs training method for mitigating pattern collapse provided in the embodiments of the present invention.

[0042] In one possible implementation, the aforementioned computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, and an optical data storage device, etc.

[0043] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

[0044] The above descriptions are merely some embodiments of the present invention. Those skilled in the art can make various modifications and improvements without departing from the inventive concept of the present invention, and these all fall within the scope of protection of the present invention.

Claims

1. A method for training GANs to mitigate pattern collapse, characterized in that, Includes the following steps: Step 1: Sample the data images of the real dataset, and then use the generator of the Generative Adversarial Network (GAN) to randomly simulate and generate negative samples to generate the original training positive and negative sample pairs. Step 2: Crop the original training positive and negative sample pairs to generate multiple sets of original image patches of different sizes. Step 3: Use a pre-trained feature extraction network to extract image features from the original image patch to obtain the corresponding feature map patch, and then input it into the generator of GANs. Step 4, set the objective function of GANs during training to min G [max D V(D,G)+λ p L P (G)+λ f L f (G)], the generator and discriminator of GANs are alternately trained and optimized based on the objective function. The training stops when the preset convergence condition is met, and a trained target image generator is obtained. Where G represents the generator, D represents the discriminator, and λ p , λ f The penalty term L for the patch sample P (G) and the penalty term L for the patch sample of the specified network layer. f (G) weight coefficients, V(D,G) is the basic objective function of generative adversarial networks, which is used to make the generator generate as realistic image data as possible, and to make the discriminator more accurate in distinguishing whether the input image comes from reality or is generated by the generator. Penalty item for patch sample This represents the penalty term of the k-th convolutional layer of the generator; Penalty item for patch samples of specified network layers Where m represents the layer number of a specified convolutional layer in the feature extraction network. This indicates the penalty term for convolutional layer m in the specified set of convolutional layers; Wherein, for any k-th convolutional layer, its corresponding penalty term The expression is: Where μ and σ represent the average value and standard deviation of the output feature map of the convolutional layer, respectively. The superscript i is the identifier of the image patch, the subscripts g and r are used to distinguish between the generated image and the real image, respectively, and n represents the number of image patches.

2. The method as described in claim 1, characterized in that, In step 2, when cropping the original training positive and negative sample pairs, they are sliced ​​into squares of multiple sizes.

3. The method as described in claim 1, characterized in that, The expression for the basic objective function V(D,G) is: Where E[·] represents the expectation, x represents the real image, G(·) represents the generator output, D(·) represents the discriminator output, z represents the input random noise, and p z (z) represents the distribution of the random noise vector, and Pdata(x) represents the distribution of the real image x.

4. The method as described in claim 1, characterized in that, The feature extraction network is a VGG series network.

5. A computer device, comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 4.

6. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method according to any one of claims 1 to 4.

Images