Image Generation Method Based on Gradient Noise Suppression Generative Adversarial Network

Abstract

The present invention discloses an image generation method based on a gradient noise suppression generative adversarial network, the method comprising the following steps: obtaining a training set for a positive sample of a discriminator; generating an image set as a negative sample; training a gradient noise perceptron; setting training parameters in GNS-GAN adversarial learning, calculating and updating the weights of the discriminator and the generator; finally, using the generator to generate an image and perform pixel value scale exchange to complete the generation of the final image. The advantages of the present invention are: improving the quality of the generated image and the stability of the adversarial learning, effectively reducing the influence of noise in the training process through the gradient noise suppression technology; reducing the training time and computing resource consumption, avoiding the complex forward and reverse diffusion steps in the traditional diffusion model; improving the network structure, making the number of parameters of the generator and the discriminator smaller, and improving the training efficiency of the model.

Description

Technical Field

[0001] The present invention relates to the technical field of artificial intelligence image generation, and in particular to an image generation method based on a gradient noise suppression generative adversarial network. Background Art

[0002] Generative artificial intelligence is one of the important basic technologies for the next generation of new artificial intelligence and general artificial intelligence. In generative artificial intelligence, generative adversarial networks and diffusion models occupy an important position. Both generative adversarial networks (GANs) and diffusion models (DMs) can be used in high-quality image generation technology, but compared with generative adversarial networks, diffusion models have the disadvantages of difficult to change the dimension of input training samples, low sample sampling efficiency, poor fitting likelihood function, no latent space encoding ability, weak data generalization ability, particularly long training time, and high training hardware requirements. Therefore, it has important practical significance and application value to generate images quickly, flexibly and with quality assurance.

[0003] Generative adversarial networks are a type of generative model with broad application value that can be naturally extended to a variety of more specific downstream tasks, such as super-resolution image or video generation tasks with greater image realism, image style transfer and video generation, segmentation of regions of interest in images or videos, and zero-shot or small-shot learning techniques for complex military or civilian environments.

[0004] Due to the adversarial learning technology used for generative adversarial network training, its stability has always been a difficulty in generative adversarial network training. At present, most research work on adversarial learning stability focuses on network structure, optimization function and game ability balance. However, gradient noise is also an important factor affecting the stability of adversarial learning.

[0005] Diffusion model (DM) is a classic image generation technique, which usually includes two stages: forward diffusion and reverse diffusion. The forward diffusion process degrades the input image into random noise by gradually adding Gaussian noise, while the reverse diffusion process reconstructs the original image by removing noise layer by layer. Although this method performs well in generating high-quality images, it has some obvious disadvantages:

[0006] Many training steps: The forward diffusion process from the input image to random noise requires hundreds to thousands of steps, and the reverse diffusion process also requires multiple steps, resulting in low training efficiency and long training time.

[0007] Complex network structure: A large number of network layers are required to handle the process from noise to image, which increases the complexity of the network and the difficulty of training.

[0008] Poor flexibility: The image size must be consistent, which limits the flexibility of the diffusion model in processing images of different sizes.

[0009] Slow inference speed: Due to the large number of network layers, the inference speed is slow, especially on edge computing devices.

[0010] Generative Adversarial Network (GAN) is a widely used generative model that generates high-quality images through an adversarial learning process. The core of GAN lies in the adversarial training of the generator (G) and the discriminator (D). Although GAN performs well in generating images, it also has some disadvantages:

[0011] Stability issues: The stability of adversarial learning is the main challenge in training GANs. Common problems include gradient disappearance and unstable training process.

[0012] Limited feature expression capabilities: Early GAN models such as DCGAN performed poorly in terms of the details and diversity of generated images. The loss function is usually based on JS divergence, which easily leads to gradient disappearance and affects the quality of generated images.

[0013] The noise-resistant learning and coalition game generative adversarial network (ANL-CG GAN) technology improves the stability of adversarial learning by introducing noise-resistant learning and coalition game modules. The noise-resistant learning module processes the positive sample loss function, and the coalition game module improves the stability of adversarial learning by enhancing the game ability of the generator.

[0014] However, these methods also have disadvantages:

[0015] Limited noise-resistant learning: Only the positive sample loss function of the D network is subjected to noise-resistant processing, which has limited effect.

[0016] Increased complexity: The coalition game module increases the complexity of adversarial learning, resulting in an increase in the number of network parameters.

[0017] Loss function problem: The loss function is still based on JS divergence, which has problems of gradient vanishing and limited feature expression ability, affecting the stability of adversarial learning and the quality of generated images. Summary of the invention

[0018] In view of the defects of the prior art, the present invention provides an image generation method based on a gradient noise suppression generative adversarial network. In the new generative adversarial network, by analyzing the gradient noise of the discriminator and the generator, a stochastic differential equation related to the corresponding gradient noise is established, and then the gradient noise distribution hypothesis is used to analyze the factors affecting the stability of adversarial learning, and a gradient noise suppression adversarial learning method for the discriminator and the generator is designed to complete the adversarial training of GNS-GAN, thereby improving the stability of adversarial learning and the quality of generated images, and having the advantages of simple and fast training for diffusion models and higher data generalization ability.

[0019] In order to achieve the above invention object, the technical solution adopted by the present invention is as follows:

[0020] A method for generating an image based on a gradient noise suppression generative adversarial network comprises the following steps:

[0021] S1. Obtain a training set X, where the training set X includes at least one person image or other color image as a positive sample for discriminator training.

[0022] S2. Use the trained generative model to generate an image set G(Z), which is used as negative samples for discriminator training.

[0023] S3, use the training data set X and the generated image set G(Z) to train the gradient noise perceptron D * , the network is used to perceive gradient noise.

[0024] S4. Perform GNS-GAN adversarial learning in image generation, including the following sub-steps:

[0025] S41. Set the learning rate, momentum factor, batch size, constraint weights, and pre-trained classifier N during training. * , this classifier has the same network structure as D;

[0026] S42, input the training data sample into D * In , the gradients of positive and negative samples are calculated, and the global average gradient is obtained;

[0027] S43, sampling batch training image samples and random vector samples for subsequent gradient calculation;

[0028] S44, using the batch of positive sample images and the generated samples, calculating the statistics of the gradient noise;

[0029] S45, calculating the distribution of batch gradient noise according to the probability density of the gradient noise;

[0030] S46, updating the discriminator network D according to the weight calculated for each sample;

[0031] S47, updating the discriminator network D using the discriminator loss function and the optimization method;

[0032] S48, updating the generator network G using the generator loss function and the optimization method;

[0033] S49, if the generator network G has not converged, then jump to step S43 to continue updating until it converges;

[0034] S410, storing the GNS-GAN model composed of the trained discriminator network D and the generator network G;

[0035] S5. In the process of generating images, a random generation method is used to generate random vectors of the image, and the generator obtained by the GNS-GAN model is used to generate the image, and the pixel value scale of the image is exchanged to complete the generation of the final image.

[0036] Furthermore, in step S42, the global average gradient of the positive sample is calculated, and the formula is as follows:

[0037]

[0038] Calculate the global average gradient of negative samples, the formula is as follows:

[0039]

[0040] In the formula, is the global average gradient of positive samples;

[0041] is the global average gradient of negative samples;

[0042] n is the total number of positive samples in the training data set;

[0043] min(0,-1+D * (x i )) is the discriminator for sample x i The difference between the output and the target value -1;

[0044] D * (x i ): Discriminator D * For the input sample x i The predicted value of

[0045] Represents the discriminator parameter θ D The gradient of

[0046] min(0,-1-D * (G(z i ))) represents the discriminator's generation of the image G(z i ) and the target value -1.

[0047] D * (G(z i )) represents the discriminator D * For the generated image G(z i )’s predicted value;

[0048] Represents the discriminator parameter θ D gradient.

[0049] Furthermore, in step S45, the distribution of batch gradient noise is calculated according to the probability density of gradient noise, and the formula is as follows:

[0050]

[0051] In the formula, δ D*，k (θ D* ;x i ) indicates that in the kth iteration, the discriminator D* targets the real sample x i The calculated gradient noise;

[0052] δ D*，k (θ D* ; G(z i )) indicates that in the kth iteration, the discriminator D* generates samples G(z i ) The calculated gradient noise;

[0053] μ 1 Represents the real sample x i The mean of the calculated gradient noise;

[0054] μ 2 Represents the generated sample G(z i )The mean of the gradient noise calculated;

[0055] Represents the real sample x i The variance of the calculated gradient noise;

[0056] Represents the generated sample G(z i )The variance of the gradient noise calculated;

[0057] f(δ D*，k (θ D* ;x i )) and f(δ D*，k (θ D* ; G(z i ))) represent the probability density functions of the gradient noise calculated for the real samples and the generated samples respectively;

[0058] and Represents the standardized coefficients of the probability density function of the normal distribution.

[0059] Furthermore, in S46, the discriminator network D is updated according to the weight calculated for each sample, and the specific formula is as follows:

[0060]

[0061] Obtaining weight and And let γ i =β i , where α i and β i Represent the real sample x i And generate samples G(z i ) in the current batch.

[0062] Furthermore, where α i is the discriminator positive sample real image optimization function gradient contribution weight, β i is the discriminator negative sample generation image sample optimization function gradient contribution weight, γ i =β i is the LG function gradient contribution weight.

[0063] Furthermore, the discriminator loss function L in S47 and S48 is D And the generator loss function As follows;

[0064]

[0065] The optimization method is the RMSProp gradient descent method.

[0066] Further, using The generator network G is updated twice using the RMSProp gradient descent method.

[0067] Preferably, the gradient noise sensor D* uses one of a convolutional neural network discriminator, a visual Transformer discriminator, and a DenseNet discriminator instead of a residual discriminator.

[0068] Preferably, the network structure of the generator G and the discriminator D is designed to be one of a convolutional network, a residual network, a visual Transformer or a DenseNet to adapt to the perception and adaptive suppression of gradient noise.

[0069] Preferably, the gradient noise hypothesis distribution in step S45 is replaced by a partial tail distribution for calculating the weight.

[0070] Compared with the prior art, the advantages of the present invention are:

[0071] 1. Improve sampling efficiency: Compared with the DDPM image generation technology, the GNS-GAN model of the present invention does not require forward diffusion and reverse diffusion processes, and generates images by gradually upsampling random noise, which greatly reduces the computational complexity and training time, and improves the sampling efficiency of image generation.

[0072] 2. Reduce the number of model parameters: The generator and discriminator use customized residual networks, and the number of network layers is generally within 15. Compared with the hundreds to thousands of layers of networks in DDPM, this reduces the number of model parameters and reduces the learning difficulty and training complexity of the network.

[0073] 3. Improve generation flexibility: The dimension of the random noise vector used by the generator (64 to 128 dimensions) is much smaller than the image dimension, which improves the flexibility of image data generation and effectively reduces the consumption of computing resources.

[0074] 4. Enhanced image diversity: Compared with DCGAN, the customized identity mapping and variable-size mapping residual blocks are used in combination, and the number of network layers is increased to about 15, thereby improving the network's feature expression ability and the diversity of generated images.

[0075] 5. Improve the ability to express details: The customized residual block and the deeper network structure (about 15 layers) combined with the number of channels in each layer of the network (128 to 512 channels) enhance the ability to express details of the generated images, making the details of the generated images richer and more realistic.

[0076] 6. Stable adversarial learning: Using the hinge loss function with gradient noise suppression ability, optimizing the reverse KL divergence, controlling the impact of gradient noise, solving the gradient vanishing problem caused by JS divergence, and improving the stability of adversarial learning and the quality of generated images.

[0077] 7. Comprehensive gradient noise suppression: Compared with ANL-CG GAN, this paper performs comprehensive gradient noise suppression on the positive and negative sample loss functions of the discriminator, as well as the loss function of the generator, and combines the weighted constraints of the batch mean gradient and the global gradient mean to further enhance the stability of adversarial learning.

[0078] 8. Simplify the training process: In order to simplify the complexity of adversarial learning, the present invention does not use a coalition game module and only includes a generator and a discriminator, which reduces the training complexity while maintaining good generation performance.

[0079] 9. Improve overall performance: By combining the hinge loss function with gradient noise suppression capability and a customized residual block, the stability of adversarial learning and the quality of generated images are improved, making the generated images excellent in quality and diversity. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] Figure 1 It is a framework diagram of the GNS-GAN image generation technology according to an embodiment of the present invention. DETAILED DESCRIPTION

[0081] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and examples.

[0082] The present invention provides an image generation method based on a gradient noise suppression generative adversarial network, comprising the following steps:

[0083] S1. Obtain a training set X, where the training set X is a person image or other color image;

[0084] S2. Use the trained generative model to obtain the generated image set G(Z);

[0085] S3, using the training data set X and the generated image set G(Z) to train a gradient noise perceptron D*;

[0086] S4, such as Figure 1 As shown in Figure 1, GNS-GAN adversarial learning in image generation includes:

[0087] S41. Input training dataset X, learning rate η, momentum factors βA and βB, batch size m, constraint weight λ, and pre-trained classifier N* (N1 and N2 are replaced by network D*, which have exactly the same network structure as D).

[0088] S42, all training data samples Input into D* and obtain the global average gradient corresponding to the positive and negative optimization functions of D*:

[0089]

[0090] S43. Sampling batch training image samples Sampling batches of random vector samples

[0091] S44. Using batch positive sample images and batch generation samples Calculate batch gradient noise δ D*，k (θ D* ;x i ) and δ D*，k (θ D* ; G(z i )).

[0092] S45, Basis and Find the batch gradient noise probability density function f(δ D*，k (θ D* ;x i )) and f(δ D*，k (θD* ; G(z i ))).

[0093] S46, Basis and Find the weight and And let γ i =β i .

[0094] S47, Utilization And RMSProp gradient descent method to update the D network.

[0095] S48, Utilization And RMSProp gradient descent method to update the G network.

[0096] S49, Utilization The G network is updated twice using the RMSProp gradient descent method.

[0097] S410. If the G network has not converged at this time, jump to step S43, otherwise output the trained generator G.

[0098] S411. Store the GNS-GANs consisting of the trained discriminator D and the trained generator G.

[0099] S5. Generate an image random vector by using a random generation method. Take the image random vector as input, use the generated image obtained in step S11, perform pixel value scale exchange on the generated image, and complete the image generation.

[0100] By establishing a stochastic differential equation related to gradient noise, factors affecting the stability of adversarial learning are found, thereby improving the stability of adversarial learning and improving the quality of generated images. According to the gradient descent of the generator G and the discriminator D of GAN, start from the gradient descent ordinary differential equation without gradient noise. By introducing additive gradient noise, the stochastic differential equation of GAN is constructed to help the model jump out of the local optimal solution, prevent overfitting, and improve the robustness of the model to noise and interference, and find out the factors affecting the stability of adversarial learning. Reasonable assumptions are made on the distribution of additive gradient noise, and gradient noise is suppressed by using gradient noise probability and gradient mean constraint terms to improve the stability of adversarial learning and the quality of generated images.

[0101] The step S45 is specifically as follows:

[0102] Where D* is the pre-trained discriminator D* that is fully trained with the training dataset image x as the positive sample and the generated dataset image G(z) as the negative sample, which can indirectly characterize the gradient sample distribution. D*，k (θ D*; x)) and f(δ D*，k (θ D* ; G(z))) respectively represent the probability value of the gradient of the training sample x or the generated sample G(z) on the D* network in the probability density function of the random gradient noise δ for any k-th batch in the current training cycle.

[0103] When training a neural network, adding a noisy probability density function to the gradient calculation of any k-th batch can help the neural network better explore the parameter space during training, thereby improving the robustness and generalization ability of training.

[0104] In step S46: i YesLD + Function (discriminator positive sample real image optimization function) gradient contribution weight, β i YesLD - Function (discriminator negative sample generation image sample optimization function) gradient contribution weight, γ i =β i is the LG function gradient contribution weight.

[0105] α i and β i The weight calculation can reward more frequently occurring gradient samples. In gradient descent, high-frequency gradient samples indicate that they are more likely to appear in the gradient sample distribution and contribute more to gradient descent.

[0106] The specific test results of the method of the present invention are as follows:

[0107] Under the conditions that the gradient descent method is RMSProp, the momentum factor βA=0.9, βB=0.0, the learning rate η=0.0002, the batch size m=64, the D network is trained once, and the G network is trained twice, step S4 is entered for training. The FID convergence curve of GNS-GAN in the training data set used for testing can be relatively smooth, and the convergence effect is the best. This shows that the generator converges to the optimal solution well during the learning process, the quality of the generated image is relatively stable, and the difference between the feature distribution and the real image is small. At the same time, it also shows that the adversarial learning method of GNS-GAN is indeed effective, with good results for convergence stability and convergence results, and can improve the stability of adversarial learning.

[0108] In the training test using the color image dataset, the IS index value of GNS-GAN is relatively high, which means that the images generated by GNS-GAN are more diverse and of higher quality, that is, the generator performs well, which can well reflect the diversity advantage of the GSN-GAN model in generating images. At the same time, the FID index value of GNS-GAN is relatively low, and the range of improved FID value is relatively small, which further reflects that GNS-GAN has better ability to simulate the distribution of training datasets.

[0109] In the training test using the face image dataset, the IS / FID value of GNS-GAN is relatively high, which means that the generator performs well in terms of the quality and diversity of the generated images, and the feature distribution of the generated images is close to that of the real images. It can also be intuitively seen from the face generated images obtained by GNS-GAN that the hair texture, facial contour and other features of the face in the generated images are better preserved, which is more in line with human visual habits.

[0110] The method according to the present invention described above can be implemented in hardware, firmware, or as software or computer code that can be stored in a recording medium (such as a CD ROM, RAM, floppy disk, hard disk or magneto-optical disk), or as computer code that is originally stored in a remote recording medium or a non-temporary machine-readable medium downloaded through a network and will be stored in a local recording medium, so that the method described herein can be stored in such software processing on a recording medium using a general-purpose computer, a dedicated processor or programmable or dedicated hardware (such as an ASIC or FPGA). It can be understood that a computer, a processor, a microprocessor controller or programmable hardware includes a storage component (e.g., RAM, ROM, flash memory, etc.) that can store or receive software or computer code, and when the software or computer code is accessed and executed by a computer, a processor or hardware, the image generation method based on a gradient noise suppression generative adversarial network described herein is implemented. In addition, when a general-purpose computer accesses the code for implementing the processing shown here, the execution of the code converts the general-purpose computer into a dedicated computer for executing the processing shown here.

[0111] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the implementation methods of the present invention, and should be understood that the protection scope of the present invention is not limited to such special statements and embodiments. Those skilled in the art can make various other specific variations and combinations that do not deviate from the essence of the present invention based on the technical revelations disclosed by the present invention, and these variations and combinations are still within the protection scope of the present invention.

Claims

1. An image generation method based on a gradient noise suppression generative adversarial network, characterized in that: The following steps are involved: S1. Obtain a training set X, wherein the training set X includes at least one person image or other color image as a positive sample for discriminator training; S2. Generate an image set G(Z) using the trained generative model. G(Z) is used as negative sample for discriminator training. S3, use the training data set X and the generated image set G(Z) to train the gradient noise perceptron D * , the network is used to perceive gradient noise; S4. Perform GNS-GAN adversarial learning in image generation, including the following sub-steps: S41. Set the learning rate, momentum factor, batch size, constraint weights, and pre-trained classifier N during training. * , this classifier has the same network structure as D; S42, input the training data sample into D * In , the gradients of positive and negative samples are calculated, and the global average gradient is obtained; S43, sampling batch training image samples and random vector samples for subsequent gradient calculation; S44, using the batch of positive sample images and the generated samples, calculating the statistics of the gradient noise; S45, calculating the distribution of batch gradient noise according to the probability density of the gradient noise; S46, updating the discriminator network D according to the weight calculated for each sample; S47, updating the discriminator network D using the discriminator loss function and the optimization method; S48, updating the generator network G using the generator loss function and the optimization method; S49, if the generator network G has not converged, then jump to step S43 to continue updating until it converges; S410, storing the GNS-GAN model composed of the trained discriminator network D and the generator network G; S5. In the process of generating images, a random generation method is used to generate random vectors of the image, and the generator obtained by the GNS-GAN model is used to generate the image, and the pixel value scale of the image is exchanged to complete the generation of the final image.

2. The image generation method according to claim 1, characterized in that: In step S42, the global average gradient of the positive sample is calculated, and the formula is as follows: Calculate the global average gradient of negative samples, the formula is as follows: In the formula, is the global average gradient of positive samples; is the global average gradient of negative samples; n is the total number of positive samples in the training data set; min(0,-1+D * (x i )) is the discriminator for sample x i The difference between the output and the target value -1; D * (x i ): Discriminator D * For the input sample x i The predicted value of Represents the discriminator parameter θ D The gradient of min(0,-1-D * (G(z i ))) represents the discriminator's generation of the image G(z i ) and the target value -1; D * (G(z i )) represents the discriminator D * For the generated image G(z i )’s predicted value; Represents the discriminator parameter θ D gradient.

3. The image generation method according to claim 2, characterized in that: In step S45, the distribution of batch gradient noise is calculated according to the probability density of gradient noise, and the formula is as follows: In the formula, δ D*,k (θ D* ;x i ) indicates that in the kth iteration, the discriminator D* targets the real sample x i The calculated gradient noise; δ D*,k (θ D* ; G(z i )) indicates that in the kth iteration, the discriminator D* generates samples G(z i ) The calculated gradient noise; μ1 represents the real sample x i The mean of the calculated gradient noise; μ2 represents the generated sample G(z i )The mean of the gradient noise calculated; Represents the real sample x i The variance of the calculated gradient noise; Indicates the generated sample G(z i )The variance of the gradient noise calculated; f(δ D*,k (δ D* ;x i )) and f(δ D*,k (θ D* ; G(z i ))) represent the probability density functions of the gradient noise calculated for the real samples and the generated samples respectively; and Represents the standardized coefficients of the probability density function of the normal distribution.

4. The image generation method according to claim 3, characterized in that: In S46, the discriminator network D is updated based on the weight calculated for each sample. The specific formula is as follows: Find the weight and And let γ i =β i , where α i and β i Represent the real sample x i And generate samples G(z i ) in the current batch.

5. The image generation method according to claim 4, characterized in that: where α i is the discriminator positive sample real image optimization function gradient contribution weight, β i is the discriminator negative sample generation image sample optimization function gradient contribution weight, γ i =β i is the LG function gradient contribution weight.

6. The image generation method according to claim 4, characterized in that: The discriminator loss function L in S47 and S48 D And the generator loss function As follows; The optimization method is the RMSProp gradient descent method.

7. The image generation method according to claim 6, characterized in that: use The generator network G is updated twice using the RMSProp gradient descent method.

8. The image generation method according to claim 1, characterized in that: The gradient noise perceptron D * Use one of the convolutional neural network discriminator, visual Transformer discriminator, and DenseNet discriminator instead of the residual discriminator.

9. The image generation method according to claim 1, characterized in that: The network structure of the generator G and the discriminator D is designed to be one of a convolutional network, a residual network, a visual Transformer or a DenseNet to adapt to the perception and adaptive suppression of gradient noise.

10. The image generation method according to claim 1, characterized in that: The gradient noise hypothesis distribution in step S45 is replaced by a partial tail distribution for calculating the weight.

Images