Comparative self-confrontation training method for optimizing GAN generation
By introducing comparative self-adversarial training methods and regularization of feature entropy values in GAN networks, the GAN generation process is optimized, and the problems of instability and pattern crash in GAN training are solved, and the generated image quality is significantly improved.
Patent Information
- Application Number
- CN202411899442.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-23
- Publication Date
- 2025-04-15
AI Technical Summary
The existing GAN network has problems with instability and pattern crash during training, resulting in low quality of generated images.
A contrasting self-adversarial training method for optimizing GAN generation is proposed. By compressing the generated image features and inputting it into the generator again, a self-adversarial training mode is formed, which enhances the discriminator's ability to capture fine-grained changes in the generation process, and optimizes the generator's output in combination with the regularization method of consistency of feature entropy values.
It significantly improves the stability of GAN training and the quality of generated images, solves the problems of pattern crash and instability, and provides new technical solutions for the application of generative adversarial networks.
Smart Images
Figure CN119830996A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of image processing, and in particular relates to a contrastive self-adversarial training method for optimizing GAN generation. Background Art
[0002] In the current field of image generation, the mainstream generative model GAN network has many problems, such as mode collapse, unstable training, etc. Regularization is generally used to optimize and alleviate these problems, but it cannot fundamentally solve the above problems. Based on this, the present invention explores the possibility of improving and optimizing GAN generation from the perspective of training strategy, and proposes a comparative self-adversarial training method for optimizing GAN generation. On the basis of not affecting the effect of regularization, the performance of GAN generation is further improved by optimizing the training strategy. Summary of the invention
[0003] The purpose of this invention is to design a contrastive self-adversarial training method for optimizing GAN generation, aiming to solve the instability and mode collapse problems in GAN training and improve the quality of generated images.
[0004] In order to achieve the above object, the technical solution adopted by the present invention is as follows:
[0005] Its overall structure diagram is as follows Figure 1 As shown, compared with the original GAN training, the training phase of the contrast self-adversarial training method for optimizing GAN generation of the present invention is divided into two stages. The first stage uses traditional GAN adversarial training, and the second stage uses self-adversarial training, and the two stages are performed in series. In the first stage, G(z) is obtained and the parameters of the discriminator D are updated. In the second stage, G(z) is feature compressed through an encoder and input into the generator G again to obtain G(G(z)). Then the two generated images G(z) and G(G(z)) are input into the Ds discriminator, with the purpose of constraining the generator so that the second generated image is better than the first.
[0006] A contrastive self-adversarial training method for optimizing GAN generation aims to:
[0007] Using the self-adversarial training mode, we first map the random noise z into a high-dimensional image G(z) using the generator G. At this time, we fix the generator G to train the discriminator D:
[0008]
[0009] Then G(z) is compressed to a low-dimensional E(G(z)) through a feature encoder, and the compressed E(G(z)) is input into the generator again to obtain G(G(z)). At this time, the generator G is fixed to train the discriminator Ds:
[0010]
[0011] In formula (2), the second output image G(G(z)) from the generator G is expected to be judged as a positive sample by the discriminator D, while the first output image G(z) is judged as a negative sample. scr and L Sup To train the discriminator Ds, by increasing the distance between G(z) and G(G(z)) in the feature space to help Ds distinguish their fine-grained differences, the generator G is trained at this time:
[0012]
[0013] In formula (3), the generator hopes that the first generated image G(z) is better than the second output image G(G(z)), thus forming a confrontation between the generator and the discriminator. Different from the traditional adversarial training, in this adversarial process, samples between the generated images are used for self-adversarial training without introducing external samples.
[0014] The overall flow chart is as follows Figure 2 As shown in the figure, there are three image distribution spaces, namely P real , the samples in the three distribution spaces are G1, G2 and Real. In the first stage, traditional adversarial training is used to optimize L D Minimize losses and P real The distance between them makes the generated G(z) closer to the real sample x. In the second stage, G(z) is input into the feature encoding compressor for dimensionality reduction, and then input into the generator again to output G(G(z)). Next, G(z) and G(G(z)) are input into the discriminator Ds for discrimination. The Ds discriminator is designed based on the idea of contrastive learning. The purpose is to use contrastive learning to learn the fine-grained differences between the two generated images. Since the two images are generated by the generator at different stages and the generator parameters are different, the discriminator can better distinguish the difference between the images generated under different generator parameters. At the same time, in order to distinguish between real samples and fake samples, supervised contrastive learning is added. Then the discriminator Ds will constrain the generator so that the second generated image is better than the first generated image. The goal of the generator is to make the first generated image better than the second generated image. Therefore, a new adversarial relationship is created between the generator and the discriminator Ds. By optimizing L Ds Losses narrowed and P real The distance between and P real The distribution distance between them can improve the generation effect.
[0015] Characteristic entropy consistency regularization method:
[0016] The present invention also proposes a feature entropy consistency regularization method, which first extracts the feature maps of the generated image and the real image data, then calculates their average entropy values for the feature maps, and then aligns the feature entropy values of the generated image with the entropy values of the real image data, which can further improve the quality of the generated image.
[0017] The following beneficial effects can be obtained through the above technical solution:
[0018] The method is simple and effective in design, and not only utilizes external data sets, but also fully applies the data generated by itself as input. As a general GAN training model, this method can be applied to any network structure designed based on GAN, in order to improve the problems caused by the defects of GAN itself. Experiments have shown that this method can further improve the generation effect in the generation field, especially in the task of GAN network generation.
[0019] The present invention aims to solve the instability and mode collapse problems in GAN training and improve the quality of generated images. The present invention adopts the contrastive self-adversarial (CSA) training mode, utilizes the self-competition between generated images to constrain the GAN generation process, and utilizes the contrast differences between images to enhance the discriminator's ability to capture fine-grained changes in the generation process. In addition, the feature entropy regularization (FeR) method in the present invention aims to stabilize the generation process and further improve the quality of generated images by aligning the data feature entropy of generated images with real images. Through a large number of experimental verifications, the method of the present invention significantly improves the stability and image quality of GAN training, promotes the latest progress in the field of regularization, and provides new ideas and technical solutions for the application of generative adversarial networks. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] Figure 1 This is a diagram comparing the difference between self-adversarial training strategy and traditional training.
[0021] Figure 2 It is the overall self-adversarial training strategy diagram.
[0022] Figure 3 It is the structure diagram of the discriminator Ds.
[0023] Figure 4 This is the encoder structure diagram.
[0024] Figure 5 It is the characteristic entropy value change graph.
[0025] Figure 6 This is a graph from the CIFAR-10 / 100 dataset.
[0026] Figure 7 (a) and (b) are the sampling test results on the CIFAR-10 / 100 dataset.
[0027] Figure 8 This is a sample test result graph on the CelebA-128 dataset. DETAILED DESCRIPTION
[0028] The following is combined with Figure 1-8 The present invention is further described as follows:
[0029] A contrastive self-adversarial training method for optimizing GAN generation, the method utilizes the self-adversarial learning process between the secondary generated images and the primary generated images generated by the self-adversarial training mode to further improve the optimized GAN training.
[0030] Using the self-adversarial training mode, we first map the random noise z into a high-dimensional image G(z) using the generator G. At this time, we fix the generator G to train the discriminator D:
[0031]
[0032] Then G(z) is compressed to a low-dimensional E(G(z)) through a feature encoder, and the compressed E(G(z)) is input into the generator again to obtain G(G(z)). At this time, the generator G is fixed to train the discriminator Ds:
[0033]
[0034] In formula (2), the second output image G(G(z)) from the generator G is expected to be judged as a positive sample by the discriminator D, while the first output image G(z) is judged as a negative sample. scr and L Sup To train the discriminator Ds, by increasing the distance between G(z) and G(G(z)) in the feature space to help Ds distinguish their fine-grained differences, the generator G is trained at this time:
[0035]
[0036] Here’s how:
[0037] S1: After the random noise z passes through the generator G, the output image G(z) is obtained and input into the discriminator D for discrimination with the real image, and the parameters of the discriminator D and the generator G are updated.
[0038] S2: The output image G(z) is input into the feature compression encoder for compression, and then the compressed z' is input into the generator again to obtain the output image G(G(z)). The output image G(G(z)) and the real data image y are input into the discriminator for discrimination, and the discriminator D parameters are updated;
[0039] S3: The contrastive learning discriminator discriminates the first generated image G(z) and the second generated image G(G(z)) of the input. The contrastive learning discriminator Ds learns the subtle differences between the two through the contrastive learning idea and updates the parameters of Ds.
[0040] S4: The contrastive learning discriminator Ds guides the second generated image to be closer to the distribution of the real image, while pushing the first generated image away from the real image distribution. After repeated iterations, the quality of the generated image will be improved.
[0041] Among them, Ds is a discriminator designed based on contrastive learning, and the overall structure is as follows Figure 3 As shown in the figure, random noise z is input into the generator to obtain G(z), and then it is input into the generator G again after passing through a feature compression encoder to obtain G(G(z)). G(z), G(G(z)) and the real sample x are input into the discriminator Ds. The discriminator first passes the three through an image enhancement module, then through a fully connected layer to map them to a three-feature representation, and then through a linear layer mapping to obtain h g ,h f ,h d , through h g ,h f Calculate the SimCLR loss through h g ,h f ,h d The supervised contrast loss is calculated. Through this process, the discriminator Ds guides the generator G to make the second generated image better than the first one.
[0042] The specific training methods are as follows:
[0043] Each parameter update consists of two stages. In the first stage, the generator G maps random noise into the first generated image G(z), and the discriminator D makes a judgment and updates the parameters of D. In the second stage, the first generated image is input into the generator G again after passing through the feature compression encoder to obtain G(G(z)), and both G(z) and G(G(z)) are put into the discriminator Ds for judgment, and the parameters of Ds and the generator are updated.
[0044] The model was trained using the Adam optimizer, with a learning rate of 0.0004, a batch_size of 16, and a total of 200,000 epochs.
[0045] The loss function and regularization method used in the above training method.
[0046] In the comparative experiment, we used three different regularization methods: CR, bCR, and ContraD. The loss functions used include two adversarial losses, feature entropy value L1 norm loss, contrast loss, and supervised contrast loss. In the first stage of the contrastive self-adversarial, the generative adversarial loss was used to constrain the training process between the generator G and the discriminator. In the second stage, the generative adversarial loss, L scr Contrastive loss, supervised contrast loss L Sup To constrain the discriminator. At the same time, in order to further improve the quality of the generated image, a feature entropy regularization constraint generator is proposed to make the image generated by the generator as consistent as possible with the feature entropy value of the real image. scr is the contrast loss function, as shown in formula (4), L Sup is the supervised contrast loss, as shown in formula (5).
[0047] Where L scr The loss is defined as follows:
[0048]
[0049] in They represent two different enhanced representations of the real image; h g Represents the view The projection representation in the feature space is,
[0050] The SimCLR loss function is defined as follows:
[0051]
[0052] Where L Sup The loss is defined as follows:
[0053] L Sup (Ds,h f )=λ1L FR +λ2L FF (6)
[0054]
[0055] Where D s represents the contrastive learning discriminator; v c,i , v f,i represents the view representation of the secondary image generated during adversarial training (e.g. Figure 3 shown); v f,-i represents the set of views that do not contain the i-th sample; h fRepresents the view v f , The projection representation in the feature space is,
[0056] The total loss in the second stage is L CSA =L adv +λ Sup L Sup +λ SCR L scr (9) where λ Sup and λ SCR are the adjustment parameters that control the loss function. We set λ SCR =1,λ Sup =0.5.
[0057] The characteristic entropy value is defined as follows:
[0058] First, the feature extraction network is used to extract feature maps from the image data. Next, the feature entropy value of each feature map is calculated according to the following formula:
[0059]
[0060] L fer =||fe(s(I g ))-fe(s(I x ))|| (12)
[0061] Where fe() is the feature extraction network, P ijk represents the probability distribution obtained by normalizing the feature map using the softmax function, and s() is the calculated feature entropy value. C represents the number of channels, and I g represents the generated image of the input, I x Represents the real image. F' represents the flattening of the calculated feature map value F.
[0062] Data set selection:
[0063] CIFAR-10 and CIFAR-100 are two commonly used datasets in the field of computer vision. CIFAR-10 contains 60,000 32x32 color images divided into 10 categories, with 6,000 images in each category. CIFAR-100 is similar, but contains 100 categories, with 600 images in each category. Some images are shown below. Figure 6 As shown in Figure 2, the CelebA-HQ-128 dataset includes 30,000 high-quality facial images with a resolution of 128×128 pixels.
[0064] Comparison of experimental results:
[0065] The present invention adopts two network structures. The first generator uses the SNDCGAN structure, the discriminator uses the SNDCGAN structure, and the Ds discriminator uses the SNDCGAN structure; the second generator uses the SNDCGAN, the discriminator uses the SNResNet-18 structure, and the Ds discriminator uses the SNResNet-18 structure. There are three regularization methods used, CR, bCR, and ContraD, among which CR is used to regularize the discriminator in GAN training by giving the discriminator a little perturbation operation (such as rotation, translation, adding noise, etc.). This constraint can prevent the discriminator from being too sensitive to subtle changes, resulting in unstable training. In order to balance the symmetric optimization between the generator and the discriminator, the generator also performs the same operation. ContraD proposes a contrastive learning discriminator, which improves the discrimination ability of the discriminator by learning the differences between different enhancement methods. On this basis, the present invention combines the advantages of contrastive learning discriminator and self-adversarial training, optimizes the GAN network from the perspective of training strategy, and thus improves GAN training.
[0066] From the results of the application of the present invention, when the network structure uses SNDCGAN and different regularization methods (CR, bCR, ContraD, etc.) are used, the indicators (FID, IS) on the CIFAR-10 and CIFAR-100 datasets and the CelebA-128-HQ dataset are improved, and there is also a certain degree of improvement under other conditions. As shown in Table 1 and Table 6.
[0067] Table 1 Comparison results of the method of the present invention verified on the CIFAR-10 / 100 dataset (the gray shading is the comparison result of the present invention)
[0068]
[0069]
[0070] The results are shown in Table 6.
[0071] Penalty +FeR +CSA W / o 25.4 24.7 24.3 CR 20.9 20.3 21.2 BCR 19.8 19.5 19.2 ContraD 17.2 17.0 18.5
[0072] In order to further improve the quality of GAN network generation, the feature entropy regularization method is used to constrain the generator. The results are shown in Table 2.
[0073]
[0074] At the same time, since the feature entropy regularization method proposed in the present invention will be affected by different feature extraction networks during feature extraction, a comparative experiment was conducted to verify its influence, as shown in Table 3.
[0075]
[0076] This paper proposes a contrastive self-adversarial training strategy to improve GAN, and introduces a feature entropy regularization to further improve the quality and diversity of the generator. The combination of these two is shown in Table 4.
[0077] Penalty +CSA +FeR FID IS W / o √ √ 27.665 6.011 CR √ √ 19.287 6.799 BCR √ √ 15.387 6.804 ContraD √ √ 14.309 7.232
[0078] In order to verify the impact of contrast loss on the training strategy, an ablation experiment was performed as shown in Table 5.
[0079]
[0080] This paper proposes a new contrastive self-adversarial training strategy, which uses the images generated by generators with different parameters at different stages during self-adversarial training as references to train the contrastive learning discriminator, and improves the contrastive learning discriminator's ability to distinguish images generated at different generation stages, thereby improving the quality of images generated by GANs. In addition, we introduce a feature entropy regularization method that aligns the feature entropy values of the generated images with the feature entropy values of the real images to further improve the quality of image generation.
[0081] The above are all preferred embodiments of the present invention. For ordinary technicians in this technical field, without departing from the principle of the present invention, various equivalent modifications to the present invention belong to the protection scope of the claims attached to this application.
Claims
1. A contrastive self-adversarial training method for optimizing GAN generation, characterized by: The method further improves and optimizes GAN training by using a self-adversarial learning process between a secondary generated image and a primary generated image generated by a self-adversarial training mode, wherein the self-adversarial training mode is as follows: First, the random noise z is mapped into a high-dimensional image G(z) using the generator G. At this time, the generator G is fixed to train the discriminator D: Then G(z) is compressed to a low-dimensional E(G(z)) through a feature encoder, and the compressed E(G(z)) is input into the generator again to obtain G(G(z)). At this time, the generator G is fixed to train the discriminator Ds: In formula (2), the second output image G(G(z)) from the generator G is expected to be judged as a positive sample by the discriminator D, while the first output image G(z) is judged as a negative sample. scr and L Sup To train the discriminator Ds, by increasing the distance between G(z) and G(G(z)) in the feature space to help Ds distinguish their fine-grained differences, the generator G is trained at this time: In formula (3), the generator hopes that the first generated image G(z) is better than the second output image G(G(z)), thus forming an adversarial relationship between the generator and the discriminator Ds.
2. A contrastive self-adversarial training method for optimizing GAN generation according to claim 1, characterized in that: The specific method of the contrastive self-adversarial training is as follows: S1: After the random noise z passes through the generator G, the output image G(z) is obtained and input into the discriminator D to discriminate with the real image, and the parameters of the discriminator D and the generator G are updated; S2: The output image G(z) is input into the feature compression encoder for compression, and then the compressed z' is input into the generator again to obtain the output image G(G(z)). The output image G(G(z)) and the real data image y are input into the discriminator for discrimination, and the discriminator D parameters are updated; S3: The contrastive learning discriminator discriminates the first generated image G(z) and the second generated image G(G(z)) input. The contrastive learning discriminator Ds learns the subtle differences between the two through contrastive learning ideas and updates the parameters of Ds. S4: The contrastive learning discriminator Ds guides the second generated image to be closer to the distribution of the real image, while pushing the first generated image away from the real image distribution. After repeated iterations, the quality of the generated image will be improved.
3. The method for optimizing the contrastive self-adversarial training of GAN generation according to claim 1, characterized in that: Each parameter update includes two stages. In the first stage, the generator G maps random noise into the first generated image G(z), and the discriminator D makes a judgment and updates the parameters of D. In the second stage, the first generated image passes through the feature compression encoder and is input into the generator G again to obtain G(G(z)). Both G(z) and G(G(z)) are put into the discriminator Ds for judgment, and the parameters of Ds and the generator are updated.
4. The method for optimizing the contrastive self-adversarial training of GAN generation according to claim 1, characterized in that: The generator G is a SNDCGAN network structure. First, the input random noise is mapped to a high-dimensional space through a fully connected layer, and then enters the upsampling stage. The output of the fully connected layer is normalized to help stabilize the training process. Through a multi-layer transposed convolutional network, the feature map is gradually upsampled to generate a high-resolution output that gradually approaches the real image; finally, the Tanh activation function is used to limit the output pixels to the range of [-1,1]; the encoder compresses the high-dimensional image data again and ensures that the main components of the image are retained as much as possible. It consists of two parallel feature extraction branches, one for processing low-scale features and the other for processing high-scale features. Each branch is composed of a convolutional layer, and each branch is connected to a self-attention module to obtain the global dependencies of different positions and improve the expression of features. Then, the low-scale and high-scale features after self-attention processing are spliced together to form a feature representation and output through a fully connected layer; the discriminator adopts two structures, SNDCGAN and SNResNet-18.
5. The method for optimizing the contrastive self-adversarial training of GAN generation according to claim 1, characterized in that: The model was trained using the Adam optimizer, with a learning rate of 0.0004, a batch_size of 16, and a total of 200,000 epochs.
6. The method for optimizing the contrastive self-adversarial training of GAN generation according to claim 1, characterized in that: Use adversarial loss to constrain the optimization process of the generator and discriminator, and use L scr Loss is used to improve the discriminator's ability to distinguish between G(z) and G(G(z)), through the supervised contrast loss L Sup To help Ds distinguish different independent samples, and also to constrain the generation process through penalty constraints such as CR and BCR to improve training stability.
7. A contrastive self-adversarial training method for optimizing GAN generation according to claim 2 or 6, characterized in that: In S1 and S3, the second generated G(G(z)) and the real image y need to be input into the discriminator D for discrimination, and the parameters of the discriminator D need to be updated. At the same time, the second generated G(G(z)) and the first generated image G(z) need to be input into the contrastive learning discriminator Ds for discrimination, and L needs to be calculated. scr loss and supervised contrast loss L Sup , update the parameters of the discriminator Ds.
8. The method for optimizing the contrastive self-adversarial training of GAN generation according to claim 1, characterized in that: Three different regularization methods, CR, bCR, and ContraD, are used for comparison. The loss function includes two adversarial losses, feature entropy value L1 norm loss, contrast loss, and supervised contrast loss. In the first stage of contrastive self-adversarial, generative adversarial loss is used to constrain the training process between the generator G and the discriminator D. In the second stage, generative adversarial loss, L scr Contrastive loss, supervised contrast loss L Sup to constrain the discriminator Ds; the feature entropy value regularizes the constrained generator so that the image generated by the generator is as consistent as possible with the feature entropy value of the real image, L scr is the contrast loss function, as shown in formula (4), L Sup is the supervised contrast loss, as in formula (5); Where L scr The loss is defined as follows: in They represent two different enhanced representations of the real image; h g Represents the view The projection representation in the feature space is, The SimCLR loss function is defined as follows: Where L Sup The loss is defined as follows: Where D s represents the contrastive learning discriminator; v c,i , v f,i represents the view representation of the secondary image generated during adversarial training; v f,-i represents the set of views that do not contain the i-th sample; h f Represents the view v f , The projection representation in the feature space is, The total loss in the second stage is L CSA =L adv +λ Sup L Sup +λ SCR L scr ; (9) where λ Sup and λ SCR They are the adjustment parameters that control the loss function, setting λ SCR =1,λ Sup =0.5; The characteristic entropy value is defined as follows: First, the feature extraction network is used to extract feature maps from the image data. Next, the feature entropy value of each feature map is calculated according to the following formula: L fer =||fe(s(I g ))-fe(s(I x ))|| (12) Where fe() is the feature extraction network, P ijk represents the probability distribution obtained by normalizing the feature map using the softmax function, s() is the calculated feature entropy value; C represents the number of channels, I g represents the generated image of the input, I x Represents the real image; F' represents the flattening of the calculated feature map value F.