A method for optimizing gan-generated images based on power spectrum loss

Abstract

The application discloses a kind of GAN class generation image optimization method based on power spectrum loss, first determine the GAN class generation network corresponding to the generation image needing optimization;The power spectrum loss of generation image is calculated;Power spectrum loss is added to the original GAN class generation network in the form of confrontation loss, and the original GAN class generation network is trained again;Finally, the generation image after optimization is generated by the GAN class generation network after training again.The method of the application is aimed at the high-frequency part of the two-dimensional power spectrum curve of GAN class generation image, only adds power spectrum loss in the original confrontation loss of GAN class generator, and under the condition that other factors remain unchanged in the training process, not only can optimize the two-dimensional power spectrum curve of generation image to a certain extent, but also can improve the quality of generation image.

Description

Technical Field

[0001] This invention belongs to the technical field of artificial intelligence image generation, specifically relating to an optimization method for GAN-based image generation based on power spectral loss. Background Technology

[0002] In recent years, with the improvement of machine computing power, the application of image generation in the field of artificial intelligence has been continuously expanding, and Generative Adversarial Networks (GANs) have become the most important research and application direction in image generation. GANs consist of a generator and a discriminator, and continuously optimize the generator through adversarial learning.To improve the quality of generated images, current research has almost entirely focused on the spatial domain: for example, CycleGAN++ (Zhang Jinglei, Hou Yawei. Image-to-image translation based on improved cycle-consistent generative adversarial network. Journal of Electronics & Information Technology, 2020, 42(5): 1216-1222) improves upon CycleGAN (Zhu Junyan, Park T, Isoal P, et al. Unpaired image-to-image translation using cycle-consistent adversarial networks / / Proceedings of the International Conference on Computer Vision. Venice, Italy, 2017: 2223-2232) by removing the circular structure of CycleGAN, eliminating the cycle consistency loss, and adding a classification loss; StyleGAN2 (Karras T, Laine S, Aittala M, et al. Analyzing and improving the image quality of StyleGAN (Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Virtual, 2020: 8107-8116) addresses the issue of images generated by StyleGAN (Karras T, Laine S, Aila T. A style-based generator architecture for generative adversarial networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Long Beach, USA, 2019: 4396-4405) potentially containing noticeable water droplets.

[0003] These studies have greatly improved the spatial quality (i.e., visual effect) of GAN-generated images, but their frequency domain quality still lags significantly behind that of natural images. This paper proposes a power spectral loss mechanism based on the upward curve of the two-dimensional power spectral curve of GAN-generated images in the very high frequency (VHF) region. This approach aims to optimize the two-dimensional power spectral curve of GAN-generated images without compromising their quality, thereby improving the overall quality of GAN-generated images. Summary of the Invention

[0004] To address the issue that existing GAN-generated images neglect frequency domain generation quality, resulting in a significant upward tilt in the VHF region of the two-dimensional power spectrum curve, this invention provides an optimization method for GAN-generated images based on power spectrum loss. This method uses the binary cross-entropy of the two-dimensional power spectrum curves of the natural and generated images to characterize the power spectrum loss, and incorporates the adversarial loss inherent in GAN generators to improve the upward tilt characteristic and generation quality of the two-dimensional power spectrum curve in the VHF region.

[0005] An optimization method for GAN-based image generation based on power spectral loss, comprising the following steps:

[0006] Step (1). Determine the GAN class of generative network corresponding to the generated image that needs to be optimized;

[0007] Step (2). Calculate the power spectral loss of the generated image;

[0008] The discriminator in a GAN is essentially a binary classifier for natural and generated images, and binary cross-entropy is a commonly used loss in binary classification problems. Therefore, the binary cross-entropy of the two-dimensional power spectrum curves of the natural and generated images can be used to label the power spectrum loss, thereby improving the problem of the two-dimensional power spectrum curve of the generated image sloping upwards in the VHF region.

[0009] Step (3). Add the power spectrum loss to the adversarial loss of the original GAN-type generator network and train the original GAN-type generator network again.

[0010] The power spectrum loss obtained in step (2) is added to the expression of the adversarial loss of the original GAN-type generator network, and the original GAN-type generator network is trained again using the original training method of the GAN-type generator network.

[0011] Step (4). Generate the optimized generated image using the retrained GAN-type generative network.

[0012] Step (2). The specific method is as follows:

[0013] The power spectral loss L is labeled by calculating the binary cross-entropy of the two-dimensional power spectral curves of the natural image and the generated image.fre The specific formula is as follows:

[0014]

[0015] Where M is the number of samples (i.e., images), and the sample size is N×N, then This represents the spatial frequency, i.e., the maximum value of the horizontal axis, in a two-dimensional power spectrum curve. and Let $\mathbf{j}$ be the j-th natural sample and the j-th generated sample, respectively. Let $PS(I)$ represent the two-dimensional power spectrum curve of sample $I$, and $Norm(·)$ represent the power spectrum curve of sample $I$. i This represents the specific value at spatial frequency i after normalizing the two-dimensional power spectrum curve, i.e., mapping the energy value to [0,1].

[0016] The specific method for step (3) is as follows:

[0017] The power spectral loss L obtained in step (2) fre Adversarial loss L compared to the original generator network adv Combined, as shown in equation (2), the new adversarial loss L is calculated. G Then use L G Replace L in the original GAN-type generator network loss function expression adv That's it; everything else remains unchanged, including the training method. Then, the existing GAN-type generative network is trained again. The formula is expressed as follows:

[0018] L G =L adv +μL adv L fre (2)

[0019] In the formula, L adv The adversarial loss of the original generative network is μ, which is a fixed weight value. A large μ may cause instability in the GAN training process, and the images generated by the model at certain times during the training process will be biased towards messy noise rather than images similar to those in the training set; while a small μ has a poor improvement effect.

[0020] The beneficial effects of this invention are as follows:

[0021] This invention addresses the problem that existing GAN-type image generation networks neglect frequency domain quality, resulting in a significant upward tilt in the two-dimensional power spectrum curve of GAN-generated images in the very high frequency (VHF) region. It employs the binary cross-entropy of the two-dimensional power spectrum curves of natural and generated images to characterize the power spectrum loss, and incorporates the adversarial loss inherent in GAN-type generators, thereby improving the upward tilt characteristic of the two-dimensional power spectrum curve of such generated images in the VHF region.

[0022] The method of this invention addresses the upward tilt of the VHF portion of the two-dimensional power spectrum curve of GAN-generated images. It adds a power spectrum loss to the original adversarial loss of the GAN generator, while keeping other factors unchanged during training. This not only optimizes the two-dimensional power spectrum curve of the generated image to a certain extent, but also improves the quality of the generated image. Attached Figure Description

[0023] Figure 1 This is a structural diagram of a general GAN ​​generator used in an embodiment of the present invention;

[0024] Figure 2 This is a two-dimensional power spectrum curve used to illustrate an embodiment of the present invention;

[0025] Figure 3 Two-dimensional power spectrum curves are provided for the natural image, the image generated by the original SAGAN generation network, and the image generated by the SAGAN generation network after adding power spectrum loss. Detailed Implementation

[0026] The invention will be further described below with reference to the accompanying drawings.

[0027] like Figure 1 As shown in the figure, G is the generator, D is the discriminator, Z is the input of the generation unit, G(Z) is the output of the generator, X is the real sample, and the discriminator loss function in the figure represents the adversarial loss. The specific steps of the method of the present invention are as follows:

[0028] Step (1). Determine the GAN class of generative network corresponding to the generated image that needs to be optimized;

[0029] Step (2). Calculate the power spectral loss of the generated image;

[0030] The discriminator in a GAN is essentially a binary classifier for natural and generated images, and binary cross-entropy is a commonly used loss in binary classification problems. Therefore, we use the binary cross-entropy of the two-dimensional power spectrum curves of the natural and generated images to label the power spectrum loss, thereby improving the problem of the two-dimensional power spectrum curve of the generated image sloping upwards in the VHF part.

[0031] To illustrate step (2), the specific method is as follows:

[0032] The power spectral loss L is labeled by calculating the binary cross-entropy of the two-dimensional power spectral curves of the natural image and the generated image. fre The specific formula is as follows:

[0033]

[0034] Where M is the number of samples (i.e., images), and the sample size is N×N, then For example, the spatial frequency in a two-dimensional power spectrum curve is the maximum value of the horizontal axis. Figure 2 The value is 720, which is dimensionless and varies with the sample size. and Let $\mathbf{j}$ be the j-th natural sample and the j-th generated sample, respectively; $PS(I)$ represents the two-dimensional power spectrum curve of sample $I$, as shown in the figure. Figure 2 As shown, the vertical axis represents the magnitude of energy, and the horizontal axis represents the magnitude of spatial frequency; a larger value indicates a larger quantity. (Norm(·)) i This represents the specific value at spatial frequency i after normalizing the two-dimensional power spectrum curve, i.e., mapping the energy value to [0,1]. For example: for Figure 2 First, the vertical coordinates are linearly mapped from 0 to 340 to [0,1]. At this point, the points corresponding to the original spatial frequencies i from 0 to 720 become the corresponding specific values ​​in [0,1] after mapping.

[0035] Step (3). Add the power spectrum loss to the adversarial loss of the original GAN-type generator network and train the original GAN-type generator network again.

[0036] Add the power spectrum loss obtained in step (2) to the expression of the adversarial loss of the original GAN-type generator network, while keeping the training method of the original generator network unchanged.

[0037] Specifically, this manifests as the power spectral loss L obtained in step (2) being... fre Adversarial loss L compared to the original generator network adv Combined, as shown in equation (2), the new adversarial loss L is calculated. G Then use L G Replace L in the original generator network loss function expression adv That's it; everything else remains unchanged, including the training method. Then, the existing GAN-type generative network is trained again. The formula is expressed as follows:

[0038] L G =L adv +μL adv L fre (2)

[0039] In the formula, L adv For the adversarial loss of the original generative network, μ is a fixed weight value (for SAGAN, the optimal weight value of μ obtained through experiments is 0.1). A larger μ may cause instability in the GAN training process, and the images generated by the model at certain times during the training process will be biased towards messy noise rather than images similar to those in the training set; while a smaller μ has a poor improvement effect.

[0040] For example, the loss function of a typical GAN-type generative network includes two parts: adversarial loss and generative loss. This invention uses the adversarial loss (regardless of its sign) as L in the formula. adv Calculate L according to the above formula G Then, replace the original losses from the confrontation, with everything else remaining unchanged;

[0041] Meanwhile, adding power spectral loss can improve the upward tilt of the VHF portion of the generated image's two-dimensional power spectral curve, but the stability of the GAN training process also needs to be considered. When μL adv A value that is too large, such as close to 1, may cause instability in the GAN training process; when μL is too large... adv When the value is too small, such as close to 0.01 or even smaller, the improvement effect is relatively poor. This is because the above formula differs from L. G =L adv +λL fre While mathematically similar, an excessively large λ causes GAN-type generative networks to overemphasize the frequency domain loss in the generated image, leading to unstable generation quality; conversely, an excessively small λ fails to achieve significant results. Therefore, vL is used... adv Replacing λ, through the original adversarial loss L adv To adjust.

[0042] Step (4). Generate the optimized generated image using the retrained GAN-type generative network.

[0043] The following uses SAGAN as an example to demonstrate the experimental results. A comparison of the two-dimensional power spectrum curves of the optimized image generated by the SAGAN generator network with the original SAGAN generator network (after incorporating power spectral loss) is shown. Each curve is obtained by averaging 1000 images. Figure 3 As shown, it can be observed that the power spectral loss improves the upward tilt of the original SAGAN-generated image overall, making it similar to the smooth descent of the real image.

[0044] It should be noted that the above embodiments can be freely combined as needed. The above description is only a detailed explanation of the preferred embodiments and principles of the present invention, but is not intended to limit the present invention. For those skilled in the art, there will be changes in the specific implementation methods based on the ideas provided by the present invention, and these changes should also be considered within the scope of protection of the present invention.

Claims

1. A method for optimizing GAN-based generated images based on power spectral loss, characterized in that, The steps are as follows: Step (1). Determine the GAN class generation network corresponding to the generated image that needs to be optimized; Step (2). Calculate the power spectrum loss of the generated image; The binary cross entropy of the two-dimensional power spectrum curve of the natural image and the generated image is used to mark the power spectrum loss; Step (3). Add the power spectrum loss to the original GAN class generation network adversarial loss, and train the original GAN class generation network again; The power spectrum loss obtained in step (2) is added to the expression of the original GAN class generation network adversarial loss, and the original GAN class generation network is trained again. The training adopts the original training method of the GAN class generation network; Step (4). Generate an optimized generated image through the GAN class generation network after retraining; The specific method of step (2) is as follows: Computing a binary cross-entropy of a two-dimensional power spectrum curve of a natural image and a generated image to label a power spectrum loss , specifically as follows: in The sample size is . ,but This represents the spatial frequency, i.e., the maximum value of the horizontal axis, in a two-dimensional power spectrum curve. and The first Zhang's natural samples and generated samples, Indicates sample Two-dimensional power spectrum curve, This indicates that a normalization operation is performed on the two-dimensional power spectrum curve, that is, the energy value is mapped to... The subsequent spatial frequency is The specific value at time.

2. The method of claim 1, wherein the method is based on a power spectral loss of a GAN class generated image. The specific method of step (3) is as follows: The power spectrum loss obtained in step (2) is replaced by The original generation network loss In combination, as shown in formula (2), a new adversarial loss is calculated ; Then replace in the original GAN type generation network loss function expression with , other contents remain unchanged, and then the original GAN type generation network is trained again; The formula expression is as follows: （2） In the formula, is the adversarial loss of the original generation network, is a fixed weight value.

Images