A gan class generation image optimization method based on high-frequency feature loss

Abstract

The application discloses a GAN class generated image optimization method based on high-frequency feature loss, first, determining the GAN class generated network corresponding to the generated image needing to be optimized; calculating the high-frequency feature loss of the generated image; then adding the high-frequency feature loss to the original GAN class generated network; the original modified GAN class generated network is trained again; finally, the optimized generated image is generated through the GAN class generated network after the retraining. The method can obviously improve the quality and diversity of the generated image under the condition that other factors remain unchanged in the training process, and the generated image generated by the GAN class generator using Trans-conv has a large amount of chaotic high-frequency noise.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of artificial intelligence image generation, and particularly relates to a GAN type generated image optimization method based on high-frequency feature loss. BACKGROUND

[0002] In recent years, with the improvement of machine computing power, the application of image generation in the field of artificial intelligence is constantly expanding, and the generative adversarial network (GAN) has become the main research and application direction of image generation. GAN is composed of a generator and a discriminator, and constantly optimizes the generator in an adversarial learning way.To improve the quality of generated images, current researches almost stay in the spatial domain: CycleGAN++(Zhang Jinglei, Hou Yawei. Image-to-image translation based on improved cycle-consistent generative adversarial network. Journal of Electronics & Information Technology, 2020, 42(5): 1216-1222) improves the phenomenon that the generator of CycleGAN(Zhu Junyan, Park T, Isoal P, et al. Unpaired image-to-image translation using cycle-consistent adversarial networks / / Proceedings of the International Conference on Computer Vision. Venice, Italy, 2017: 2223-2232) may hide some features in the input image and restore them when outputting, cancels the ring structure of CycleGAN, removes the cycle consistency loss and adds a classification loss; StyleGAN2(Karras T, Laine S, Aittala M, et al. Analyzing and improving the image quality of styleGAN / / Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Virtual, 2020: 8107-8116) improves the problem that the images generated by StyleGAN(Karras T, Laine S, Aila T. A style-based generator architecture for generative adversarial networks / / Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Long Beach, USA, 2019: 4396-4405) may have obvious water droplets.

[0003] These studies greatly improve the quality of GAN-generated images in the spatial domain (i.e., in visual effects), but there is still a large gap in the frequency domain, especially in the high-frequency part, from natural images. GAN generators use transposed convolution (Trans-conv) to generate images that can retain more high-frequency details, so researchers often use Trans-conv as the decoding method for the generator. Based on the phenomenon that the images generated by GAN generators using Trans-conv have a large amount of chaotic noise in the high-frequency part compared to natural images, this paper proposes a high-frequency feature loss to improve the quality and diversity of generated images. SUMMARY

[0004] To address the problem of existing GAN-generated images ignoring the high-frequency part of the frequency domain, resulting in a large amount of chaotic high-frequency noise in the images generated by the generator using Trans-conv, the present application provides an optimization method for GAN-generated images based on high-frequency feature loss, which uses the distance between the high-frequency information of the very high-frequency part of natural images and generated images in the feature space to represent the high-frequency feature loss, and adds the original adversarial loss of the GAN generator to improve the quality and diversity of such generated images.

[0005] An optimization method for GAN-generated images based on high-frequency feature loss, the steps are as follows:

[0006] Step (1). Determine the GAN-generated network corresponding to the generated image that needs to be optimized;

[0007] Step (2). Calculate the high-frequency feature loss of the generated image;

[0008] Construct the high-frequency feature loss (denoted as L hf ) based on the idea of contrast loss to optimize the quality of GAN-generated images. Contrast loss can well express the matching degree between samples. After dimension reduction, i.e., feature extraction, two sample sets that are originally similar are still similar in feature space. Therefore, the distance between the high-frequency information of the very high-frequency part of the generated image and the natural image in the feature space is used to mark the high-frequency feature loss, thereby improving the problem of such generated images having a large amount of chaotic high-frequency noise.

[0009] Step (3). Add the high-frequency feature loss to the adversarial loss of the original GAN-generated network, and retrain the modified GAN-generated network;

[0010] Add the high-frequency feature loss obtained in step (2) to the expression of the adversarial loss of the original GAN-generated network, and retrain the original GAN-generated network. The training uses the original training method of the original GAN-generated network unchanged.

[0011] Step (4). The optimized generated image is generated by the GAN class generation network after being trained again.

[0012] Step (2). The specific method is as follows:

[0013] The distance of the high-frequency information of the very high frequency part of the natural image and the generated image in the feature space is calculated to mark the high-frequency feature loss L hf , and the specific formula is as follows:

[0014]

[0015] d(x i ,x i ′)=||HL(HP(x i ))-HL(HP(x i ′))||2 (1)

[0016] Wherein, x i and x′ i are natural samples and generated samples respectively, d(·) represents the Euclidean distance, HP(·) represents the high-frequency information of the sample obtained by the high-pass filter, HL(·) represents the function of mapping the input of the high-dimensional space to the output of the low-dimensional space, and M is the number of samples.

[0017] Step (3). The specific method is as follows:

[0018] The high-frequency feature loss L hf obtained in step (2) is combined with the original generated network adversarial loss L adv , as shown in formula (2), the new adversarial loss L G is calculated; then L G replaces L adv in the original generated network loss function expression, and other contents are unchanged, including the training method, etc., and then the original GAN class generation network is trained again. The formula expression is as follows:

[0019] L G =L adv +γL hf (2)

[0020] In the formula, L adv is the original generated network adversarial loss, and gamma is a fixed weight value. A large gamma may cause instability in the GAN training process, and the image generated by the model at some moments in the training process will be biased towards chaotic noise, rather than images similar to the training set; and a small gamma has a poor improvement effect.

[0021] The present application has the following beneficial effects:

[0022] This invention addresses the problem that existing GAN-type image generation networks neglect the generation quality of high-frequency components in the frequency domain, resulting in images generated by GAN-type generators using Trans-conv having a large amount of noise in the high-frequency components compared to natural images. The invention uses the distance between the high-frequency information of the very high-frequency components of natural images and generated images in the feature space to represent the high-frequency feature loss, and adds the adversarial loss originally used by GAN-type generators, thereby improving the quality and diversity of generated images.

[0023] The method of this invention addresses the problem that images generated by GAN-type generators using Trans-conv contain a large amount of messy high-frequency noise. It simply adds high-frequency feature loss to the original adversarial loss of the GAN-type generator, while keeping other factors unchanged during training, which can significantly improve the quality and diversity of the generated images. Attached Figure Description

[0024] Figure 1 This is a structural diagram of a general GAN ​​generator used in an embodiment of the present invention;

[0025] Figure 2 This embodiment of the invention illustrates the process of calculating the distance between a natural image and an image generated by the generator using Trans-conv. Detailed Implementation

[0026] The invention will be further described below with reference to the accompanying drawings.

[0027] like Figure 1 As shown in the figure, G is the generator, D is the discriminator, Z is the input of the generation unit, G(Z) is the output of the generator, X is the real sample, and the discriminator loss function in the figure represents the adversarial loss. The specific steps of the method of the present invention are as follows:

[0028] Step (1). Determine the GAN class of generative network corresponding to the generated image that needs to be optimized;

[0029] Step (2). Calculate the high-frequency feature loss of the generated image by the generative network;

[0030] A high-frequency feature loss is constructed based on the idea of ​​contrastive loss. Contrastive loss can effectively express the degree of matching between samples. Two sample sets that were originally similar remain similar in the feature space after dimensionality reduction, i.e., feature extraction. Therefore, the high-frequency feature loss is characterized by the distance between the high-frequency information of the VHF components of the natural image and the generated image in the feature space, thereby improving the quality and diversity of the generated images.

[0031] To illustrate step (2), the specific method is as follows:

[0032] The high-frequency feature loss K is labeled by calculating the distance between the high-frequency information of the VHF components of the natural image and the generated image in the feature space.hf The specific formula is as follows:

[0033]

[0034] d(x i ,x′ i )=||HL(HP(x i ))-HL(HP(x′ i ))||2 (1)

[0035] Where x i and x′ i Let HP(x) be the i-th natural sample and the generated sample, respectively. i ) and HP(x′ i ) represent the high-frequency information obtained from the i-th natural sample and the generated sample after passing through a high-pass filter, for example Figure 2 The natural sample x and the generated sample x′, along with their corresponding high-frequency information, Figure 2 In this context, DFT and IDFT represent the two-dimensional Discrete Fourier Transform (DFT) and Inverse Discrete Fourier Transform (IDFT), respectively. HL(HP(x)) denotes a function that maps a high-dimensional input to a low-dimensional output, for example... Figure 2 Dimensionality reduction of medium and high frequency information is the feature extraction operation. Figure 2 In this context, d(x,x′) represents the calculation of the Euclidean distance between natural samples and generated samples to measure the difference between the samples.

[0036] Repeat the above process, where M is the number of samples, to obtain the distance in feature space between the high-frequency information of the VHF components of the M natural images and the corresponding generated images. In L... hf In the formula, for d(x) i ,x′ i The squares are used to eliminate the influence of positive and negative differences and amplify the effect of the differences. The sum of the squares of the Euclidean distances between M samples is calculated, and finally, the sum of the squares of the distances is standardized by dividing by 2M.

[0037] Step (3). Add the high-frequency feature loss to the adversarial loss of the original GAN-type generator network, and retrain the original modified GAN-type generator network.

[0038] Add the high-frequency feature loss obtained in step (2) to the expression of the adversarial loss of the original GAN-type generative network, while keeping the training method of the original generative network unchanged.

[0039] Specifically, this manifests as the high-frequency feature loss L obtained in step (2) being... hfAdversarial loss L compared to the original generator network adv Combined, as shown in equation (2), the new adversarial loss L is calculated. G Then use L G Replace L in the original generator network loss function expression adv That's it; everything else remains unchanged, including the training method. The formula is expressed as follows:

[0040] L G =L adv +γL hf (2)

[0041] In the formula, L adv The adversarial loss of the original generative network is γ, which is a fixed weight value (taking SAGAN as an example, the optimal weight value obtained through experiments is 1e-5). A large γ may cause instability in the GAN training process, and the images generated by the model at certain times during the training process will be biased towards messy noise rather than images similar to the training set; while a small γ has a poor improvement effect.

[0042] For example, the loss function of a typical GAN-type generative network includes two parts: adversarial loss and generative loss. This invention uses the adversarial loss (regardless of its sign) as L in the formula. adv Calculate L according to the above formula G Then, replace the original losses from the confrontation, with everything else remaining unchanged;

[0043] Meanwhile, adding high-frequency feature loss can improve the quality and diversity of generated images, but it's also necessary to consider ensuring the stability of the GAN training process. When γ is too large, such as close to 1, it may cause instability in the GAN training process; when γ is too small, such as even smaller, the improvement effect is relatively poor. This is because an excessively large γ causes the GAN-like generative network to overemphasize the loss on high-frequency features of the generated images, leading to unstable generation quality; while an excessively small γ is unlikely to achieve significant results. Therefore, an appropriate value that can simultaneously improve the generation performance of each model is adopted, using the original adversarial loss L... adv To adjust.

[0044] Step (4). Generate the optimized generated image by retraining the generative network with high-frequency feature loss.

[0045] The following uses SAGAN as an example to demonstrate the experimental results. The image similarity evaluation index FID is used for objective evaluation, and the FID values ​​of the generated images are shown in Table 1.

[0046] Table 1 shows the FID values ​​between the original SAGAN image, the generated image after adding high-frequency loss, and the natural image.

[0047]

[0048] As can be seen from Table 1, the FID value between the image generated by adding L hf The FID value between the image generated by SAGAN after adding L hf The quality and diversity of the generated image can be improved.

[0049] It should be noted that the above embodiments can be freely combined as needed. The above only describes the preferred embodiments and principles of the present application in detail, but is not a limitation on the present application. For those skilled in the art, the specific embodiments provided by the present application can be changed according to the idea provided by the present application, and these changes should be considered as the protection scope of the present application.

Claims

1. An optimization method for GAN-based image generation based on high-frequency feature loss, characterized in that, The steps are as follows: Step (1): Determine the GAN class generator network corresponding to the generated image that needs to be optimized; Step (2): Calculate the high-frequency feature loss of the generated image; A high-frequency feature loss is constructed based on the idea of ​​contrastive loss to optimize the quality of images generated by GAN; the high-frequency feature loss is labeled by the distance between the high-frequency information of the VHF part of the generated image and the natural image in the feature space. Step (3): Add the high-frequency feature loss to the adversarial loss of the original GAN-type generator network, and retrain the original modified GAN-type generator network. Add the high-frequency feature loss obtained in step (2) to the expression of the adversarial loss of the original GAN-type generator network, and train the original GAN-type generator network again. The original training method of the original GAN-type generator network remains unchanged. Step (4): Generate the optimized generated image using the retrained GAN-type generative network; Step (2): The specific method is as follows: The high-frequency feature loss is labeled by calculating the distance between the high-frequency information of the VHF components of the natural image and the generated image in the feature space. The specific formula is as follows: （1） in, and They are natural samples and generated samples, respectively. Represents Euclidean distance. This represents the high-frequency information obtained by passing the sample through a high-pass filter. A function that represents a high-dimensional input mapped to a low-dimensional output. This represents the number of samples.

2. The optimization method for GAN-based image generation based on high-frequency feature loss according to claim 1, characterized in that, Step (3): The specific method is as follows: The high-frequency feature loss obtained in step (2) Adversarial loss compared to the original generative network Combined, as shown in Equation (2), the new adversarial loss is calculated. Then then Replace the original generator network loss function expression in That's it, other content remains unchanged, and then the original GAN-type generative network is trained again; the formula is expressed as follows: （2） In the formula, The adversarial loss of the original generator network, These are fixed weight values.

Images