A small sample image generation method fusing frequency domain information

Abstract

The application discloses a small sample image generation method fusing frequency domain information, and is used for high-quality image generation in a small sample scene. The small sample image generation method fusing frequency domain information provided by the application comprises the following steps: pre-processing training pictures, and using the pre-processed training pictures to train a discriminator network and a generator network; the discriminator network comprises a picture feature discrimination module, a high-frequency discrimination module and a discriminator network loss module; the generator network comprises a mapping generated picture module, a frequency domain skip link module and a generator network loss module; the generator network and the discriminator network are iteratively updated according to respective losses until a maximum iteration number is reached. The application can generate more diverse, more real and higher-quality pictures in a small sample scene.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer vision in the direction of artificial intelligence, and in particular to a small sample image generation method fusing frequency domain information; mainly applied to the fields of image editing, generation and image expansion enhancement. BACKGROUND

[0002] Deep generative models have made significant progress and been widely applied in computer vision tasks such as image generation, image translation, image inpainting and image dehazing since they were proposed. However, on the one hand, the success of existing generative models is largely dependent on large-scale training data. When the training data is small, the generative model is very easy to overfit, resulting in poor generation of real samples and low diversity of generated samples. On the other hand, collecting a large amount of data not only consumes a lot of resources, but even in some fields it is not achievable, such as famous paintings, medical images of rare diseases, or pictures of aerospace scenes. Therefore, it is of great application value and practical significance to study how to apply generative models, especially generative adversarial networks (GAN), to small sample scenarios with only a small number of training pictures, that is, to improve the quality of generated pictures in small sample image generation. The generated pictures not only achieve data expansion of limited samples, but also can be used to assist downstream classification, segmentation and other tasks.

[0003] The existing mainstream small sample image generation methods mainly include the following categories:

[0004] 1) using data augmentation techniques to expand training samples;

[0005] 2) using regularization techniques to constrain the discriminator network of GAN;

[0006] 3) designing new network structures and modules, such as attention mechanisms, skip activation mechanisms, etc.;

[0007] 4) using transfer learning methods to transfer the pre-trained knowledge in large-scale data (source domain) to the target domain with limited samples;

[0008] The above methods have the following problems:

[0009] 1) The data augmentation techniques in may cause the training data to be leaked, and since the training pictures are flipped, translated, etc., the original data distribution may be changed, thereby misleading the generative model to generate pictures of the changed distribution;

[0010] 2) The regularization techniques in are usually very sensitive to weight and hyperparameter adjustments, and require a lot of manual adjustments for training and model optimization on new data sets;

[0011] The new mechanism in 3) increases the parameter quantity of the network, expands the calculation amount, and increases the risk of overfitting;

[0012] The transfer learning method in 4) still needs the assistance of an additional source domain, and when there is a large inter-domain gap between the source domain and the target domain, overfitting may occur, thereby reducing the generation performance;

[0013] In addition, the existing small sample image generation method ignores an important piece of information of image generation, frequency domain information. Deep neural networks only fit the low-frequency information of the image in the process of fitting the signal, and the high-frequency information is usually ignored. High-frequency information contains the texture detail information of the picture. Ignoring this kind of information will cause the generated picture to perform poorly in details, affecting the overall authenticity of the generated picture. SUMMARY

[0014] The purpose of the present application is to overcome the deficiencies of the prior art and provide a small sample image generation method that fuses frequency domain information to improve the quality of generated pictures in a small sample scenario.

[0015] The purpose of the present application can be achieved by the following technology:

[0016] A small sample image generation method that fuses frequency domain information, comprising the following steps:

[0017] 1) Training picture preprocessing step, obtaining preprocessed training pictures, used to train the discriminator network and the generator network;

[0018] 2) Training the discriminator network step, training the discriminator network based on the preprocessed training pictures, the discriminator network comprising a picture feature discrimination module, a high-frequency discrimination module, and a discriminator network loss module;

[0019] 3) Training the generator network step, training the generator network based on the discriminator network, the generator network comprising a mapping generated picture module, a frequency domain skip link module, a high-frequency information alignment module, and a generator network loss module;

[0020] 4) Alternately training the discriminator network and the generator network until the maximum number of iterations is reached.

[0021] In step 1), the training pictures include real pictures from the training set and generated pictures from the generator network. The picture preprocessing is to resize the training pictures, randomly horizontally flip them, normalize them to have a mean of 0.5 and a variance of 0.5, and then convert them to tensor form to obtain the preprocessed training pictures;

[0022] In the step 2), the picture feature discrimination module is composed of a convolution network, a ReLU activation function, a Batch Normalization network and a down-sampling operation, and is used to discriminate whether the preprocessed training picture is the real picture or the generated picture after extracting features from the preprocessed training picture;

[0023] In the step 2), the high-frequency discrimination module uses Haar wavelet transform to extract high-frequency information from the preprocessed training picture. The Haar wavelet transform includes two mirror image operations: wavelet pooling (Pooling) and wavelet unpooling (Unpooling) operations. The wavelet Pooling operation converts the preprocessed training picture into a frequency domain representation, and the wavelet Unpooling operation reconstructs the frequency domain representation into a picture representation. The wavelet Pooling operation of the Haar wavelet transform includes four different kernels: LL T , LH T , HL T , and HH T , wherein L and H represent low-frequency and high-frequency information filters, respectively. The low-frequency information filter (L) extracts coarse-grained information such as outlines and summaries of the preprocessed training picture, and the high-frequency information filter (H) extracts fine-grained information such as edges and detailed textures of the preprocessed training picture. After extracting high-frequency information from the preprocessed training picture using the Haar wavelet transform, the high-frequency information is discriminated to determine whether the high-frequency information comes from the real picture or the generated picture, with the following discrimination constraint:

[0024]

[0025] wherein E represents expectation, min() represents minimization, HF r and HF f represent the high-frequency information of the real picture and the generated picture, respectively, D H represents a high-frequency discrimination network of the discriminator network, z represents noise randomly sampled in the Gaussian distribution, G(z) represents the generated picture obtained by the mapping generated picture module of the generator network, I represents the real picture, and represent the discrimination constraints of the high-frequency discrimination module for the generator network and the discriminator network, respectively.

[0026] In the step 2), the discriminator network loss module includes a high-frequency discrimination loss, a discriminator network adversarial loss and a reconstruction loss. The high-frequency discrimination loss is the discrimination constraint of the high-frequency discrimination module on the discriminator network the discriminator network adversarial loss L D is:

[0027] L D = -E x～I [min(0,-1+D(x))]-E x～G(z) [min(0,-1-D(x))]

[0028] wherein E represents expectation, min() represents minimization, x represents the pre-processed training picture, D(x) is the picture feature discrimination module, z represents the noise randomly sampled in the Gaussian distribution, G(z) represents the generated picture obtained by the mapping generated picture module of the generator network, and I represents the real picture. The reconstruction loss L rec is:

[0029]

[0030] wherein f represents the feature to be reconstructed in the discriminator network, x represents the pre-processed training picture, D encode (x) represents the encoder network of the discriminator network, H(·) and T(·) represent small encoders for feature extraction of features and pictures, and I represents the real picture.

[0031] In summary, the discriminator loss is generally represented as wherein L D represents the discriminator network adversarial loss, represents the discrimination constraint of the discriminator network by the high-frequency discrimination module, and L rec represents the reconstruction loss.

[0032] In the step 3), the mapping generated picture module maps the noise randomly sampled from the Gaussian distribution to obtain a generated picture through inverse convolution, upsampling, GLU, and jump link operations;

[0033] In the step 3), the frequency domain jump link module uses the Haar wavelet to extract frequency domain information of the generated picture, and transmits the extracted frequency domain information to the deep layer of the generator network in a jump link manner:

[0034]

[0035] wherein F i+1 represents the feature obtained by the (i+1) layer of the generator network, Unpooling is the wavelet Unpooling operation, represents the frequency domain information of the (i) layer of the generator network, The new features obtained after the jump link.

[0036] In the step 3), the high-frequency information alignment module aligns the high-frequency information of the real picture and the generated picture, uses the high-frequency information of the real picture as the self-supervised guidance information for the generator network to generate high-frequency information, and constrains L align is:

[0037] L align =||HF D -HF G ||1

[0038] Wherein, HF D represents the high-frequency information extracted from the real picture by the Haar wavelet, HF G represents the high-frequency information extracted from the generated picture by the Haar wavelet, and ||·||1 represents the L1 norm.

[0039] In the step 3), the generator network loss module includes a generator network adversarial loss, a high-frequency discrimination constraint of the high-frequency discriminator module on the generator network, and a high-frequency information alignment loss. The generator network adversarial loss L G is:

[0040] L G =-E z～N [D(G(z))]

[0041] Wherein, z represents the noise randomly sampled from the Gaussian distribution N, G(z) represents the generated picture obtained by the mapping generated picture module of the generator network, D is the picture feature discriminator module, and E represents the expectation.

[0042] In summary, the total loss of the generator network is Wherein L align represents the constraint of the high-frequency information alignment module, represents the discrimination constraint of the high-frequency discriminator module on the generator network, and L G represents the generator network adversarial loss.

[0043] In the step 4), the discriminator network is optimized according to the discriminator network loss module, and the generator network is optimized according to the loss calculated by the generator network loss module. The two are alternately performed until the maximum iteration number is obtained and the training is stopped.

[0044] The application discloses a small sample image generation method fusing frequency domain information.

[0045] Compared with the prior art, the application has the following beneficial effects:

[0046] First, the application fully fuses the frequency domain information of training pictures in a small sample scene, the high-frequency discrimination module can help the generation model pay more attention to the details and textures of the pictures, improve the perception of the discriminator network to the frequency domain information, and thus provide better guidance for the generator network, and help the generator network generate more realistic pictures with rich detail information.

[0047] Second, the application fully considers the problem that the deep neural network preferentially fits the low-frequency information of the picture and easily ignores the high-frequency information, the frequency domain jump link module supplements the frequency domain information to the deep network, avoids the loss of the frequency domain information, and further improves the quality of the generated pictures.

[0048] Third, the high-frequency information alignment module aligns the high-frequency information of the real pictures and the generated pictures, uses the high-frequency information of the real pictures as the self-supervised guidance information for the generation of high-frequency information by the generator network, provides guidance for the generator network containing the information of the real pictures, effectively improves the authenticity and rationality of the generated high-frequency information, and improves the fairness of the adversarial training between the generator network and the discriminator network, promotes the stability of the training, and improves the interpretability of the model.

[0049] Fourth, the application significantly improves the diversity and authenticity of the generated pictures of a plurality of data sets in a small sample scene, and performs well in a series of objective evaluation indexes such as FID, KID, IS, Precision, Recall and the like, and can realize high-quality picture generation in a limited training sample scene. BRIEF DESCRIPTION OF DRAWINGS

[0050] Figure 1 It is the overall training flowchart provided by the embodiment of the application.

[0051] Figure 2 It is the discriminator network structure schematic diagram provided by the embodiment of the application.

[0052] Figure 3 It is the generator network structure schematic diagram provided by the embodiment of the application.

[0053] Figure 4 is the overall training framework provided by the embodiment of the present application. DETAILED DESCRIPTION

[0054] The embodiments of the present application will be described in detail below with reference to the drawings, and the system practice details will also be described in detail. However, these practice details will not limit the present application to the scope of the described embodiments.

[0055] The present application is a small sample image generation method fusing frequency domain information, which fully considers the frequency domain information of limited training samples, and designs a frequency domain skip link, a high-frequency information alignment module and a high-frequency discrimination module for the generator network and the discriminator network respectively, extracts the frequency domain information of the training picture by using the Haar wavelet transform method, transmits the extracted frequency domain information to the deep layer of the generator network in a skip link manner through the frequency domain skip link module, avoids losing part of the frequency domain information due to the deepening of the generator network, aligns the high-frequency information of the real picture and the generated picture through the high-frequency information alignment module, uses the high-frequency information of the real picture as the self-supervised guidance information for the generator network to generate high-frequency information, and improves the authenticity of the high-frequency information generated by the generator network; the high-frequency discrimination network discriminates the high-frequency information of the training picture, and discriminates whether the high-frequency information comes from the real picture or the generated picture.

[0056] The generator network and the discriminator network are updated and optimized through the generator network loss and the discriminator network loss respectively, the generator network learns how to generate more real and diverse pictures, the discriminator network learns to distinguish real pictures from generated pictures, and the two are alternately trained in an antagonistic manner, and are continuously optimized until the maximum iteration number is reached, and the training is ended.

[0057] The present application is based on the training picture preprocessing step, the discriminator network training step and the generator network training step to learn on a small sample image dataset to generate pictures with authenticity and diversity.

[0058] The training process is shown in Figure 1 , the discriminator network is shown in Figure 2 , the generator network is shown in Figure 3 , and the overall framework is shown in Figure 4 .

[0059] The specific execution process of the present application is as follows:

[0060] Step 1: initialize the training parameters;

[0061] Initialize the small sample training data picture size R, the training set T, the training batch size, the maximum number of iterations T, and randomly initialize the generator network G and the discriminator network D.

[0062] Step 2: Preprocess the training pictures, which include the pictures generated by the generator network and the real pictures sampled from the training set. The preprocessing includes resizing, random horizontal flipping, normalizing to a mean of 0.5 and a variance of 0.5, and then converting to a tensor form to obtain the preprocessed training pictures.

[0063] Step 3: Train the discriminator network D, calculate the discriminator network loss according to the discriminator network loss module, and update and optimize the discriminator network using the obtained loss. The discriminator network loss module of the discriminator network includes the high-frequency discrimination loss, the discriminator network adversarial loss, and the reconstruction loss.

[0064] Calculate the high-frequency discrimination loss

[0065]

[0066] where E represents expectation, HF r and HF f represent the high-frequency information of the real picture and the generated picture respectively, D H represents the high-frequency discrimination network of the discriminator network, G(z) represents the generated picture obtained by the mapping generated picture module of the generator network, I represents the real picture, and min() represents minimization.

[0067] Calculate the discriminator network adversarial loss:

[0068] L D =-E x～I [min(0,-1+D(x))]-E x～G(z) [min(0,-1-D(x))]

[0069] where E represents expectation, x represents the preprocessed training picture, D(x) is the picture feature discrimination module, z represents the noise randomly sampled from the Gaussian distribution N, G(z) represents the generated picture obtained by the mapping generated picture module of the generator network, I represents the real picture, and min() represents minimization.

[0070] Calculate the reconstruction loss:

[0071]

[0072] wherein f represents the features that need to be reconstructed in the discriminator network, x represents the pre-processed training picture, D encode (x) represents the encoder network of the discriminator network, H(·) and T(·) represent small encoders for feature extraction of features and pictures, and I represents the real picture.

[0073] In summary, by minimizing the overall loss of the discriminator network: The discriminator network D is trained and optimized, and the parameters of the discriminator network are updated.

[0074] Step 4, training the generator network G, calculating the loss according to the generator network loss module, and updating and optimizing the generator network using the obtained loss, wherein the generator network loss module of the generator network includes the high-frequency discrimination loss, the high-frequency alignment loss, and the generator network adversarial loss.

[0075] The high-frequency discrimination loss is calculated as:

[0076]

[0077] wherein E represents expectation, HF f represents the high-frequency information of the generated picture, D H represents the high-frequency discrimination network of the discriminator network, z represents noise randomly sampled from the Gaussian distribution, and G(z) represents the generated picture obtained by the mapping generated picture module of the generator network.

[0078] The high-frequency alignment loss is calculated as:

[0079] L align = ||HF D -HF G ||1

[0080] wherein HF D represents the high-frequency information extracted from the real picture by the Haar wavelet, HF G represents the high-frequency information extracted from the generated picture by the Haar wavelet, and ||·||1 represents the L1 norm.

[0081] The generator network adversarial loss is calculated as:

[0082] L G = -E z～N [D(G(z))]

[0083] wherein z represents noise randomly sampled from the Gaussian distribution N, G is the mapping generated picture module, D is the picture feature discrimination module, and E represents expectation.

[0084] In summary, by minimizing the total loss of the generator network: The generator network G is trained and optimized, and the generator network parameters are updated.

[0085] Step 5, check the number of iterations, if the current number of iterations is greater than the set maximum number of iterations T, exit the training, otherwise add 1 to the current number of iterations, and repeat steps 2 to 7. The maximum number of iterations T set by the present application is 100000, and the model parameters are saved every 10000 times. Finally, the saved model parameters are used to generate pictures, and the generated pictures are used to calculate the indicators, supplemented to the small sample data, or used to assist the downstream classification, segmentation and other tasks.

[0086] Experimental design:

[0087] Experimental data set

[0088] The experimental data set is selected from small sample image data sets from different fields, including animation, painting, face, landscape and other sources, containing a variety of styles, containing different resolutions, and the detailed introduction of the data is shown in Table 1. The selected data set contains a very limited number, which has a great challenge for generating new pictures, but also has very strong application and research significance.

[0089] Table 1 Introduction of experimental data set

[0090]

[0091] Comparison algorithm

[0092] The present application aims at the image generation problem in the small sample scene. The comparison algorithm is the mainstream method in the current small sample image generation field, such as StyleGAN2, DiffAug, ADA, APA and FastGAN.

[0093] Evaluation index

[0094] The commonly used indicators for evaluating the authenticity and diversity of generated pictures are FID and KID. FID calculates the difference between the mean and variance statistics of the features extracted from the InceptionV3 network of the real picture and the generated picture, measures the distribution distance between the real picture and the generated picture, selects all real training pictures as the reference real distribution, generates 50000 pictures as the reference generated distribution, and inputs them into the InceptionV3 network to calculate the difference between the two statistics to obtain the FID value. The smaller the FID, the closer the distribution of the real picture and the generated picture, and the higher the quality of the generated picture.

[0095] The calculation formula of FID is:

[0096] FID = || μr -μ g || 2 +Tr(∑ r +∑ g -2(∑ r ∑ g ) 1 / 2 )

[0097] wherein μ r and μ g represent the feature mean of the real picture and the generated picture respectively, ∑ r and ∑ g represent the covariance matrix of the real picture and the generated picture respectively, and Tr represents the trace.

[0098] KID directly calculates the distance between the features obtained after the real picture and the generated picture are input into the InceptionV3 network in the kernel space to represent the distribution difference between the real picture and the generated picture, and the smaller the KID is, the closer the generated picture is to the real picture, and the higher the generation quality is.

[0099] Experimental results

[0100] Table 2 Experimental results of FID index of the present application and the comparative method on 256*256*3 data set

[0101] Datasets AFdog AFcat Panda Grupy-cat StyleGAN2 113.86 79.04 18.05 35.00 DiffAug 61.34 41.84 11.52 26.89 ADA 55.48 37.95 14.17 43.80 APA 81.16 42.60 19.21 28.10 FastGAN 52.46 33.85 9.70 25.75 Ours 47.85 31.05 8.97 24.93

[0102] Table 3 Experimental results of KID index of the present application and the comparative method on 256*256*3 data set

[0103] Datasets AFdog AFcat Panda Grupy-cat StyleGAN2 91.31 34.43 7.40 11.01 DiffAug 24.51 12.27 3.57 9.42 ADA 18.42 6.43 6.53 45.01 APA 26.42 7.97 10.80 5.53 FastGAN 18.22 4.99 1.60 3.41 Ours 13.49 2.44 0.91 3.89

[0104] Table 4 Experimental results of FID index of the present application and the comparative method on 512*512*3 data set

[0105] Datasets AnimeFace ArtPainting Moongate Flat Fauvism StyleGAN2 183.44 100.35 288.25 285.61 299.15 DiffAug 135.85 49.25 136.12 340.14 223.58 ADA 59.67 46.38 149.06 248.46 201.99 APA 58.38 47.23 193.67 233.52 197.47 FastGAN 55.87 45.06 114.79 216.27 178.42 Ours 50.19 43.13 107.13 178.10 171.95

[0106] Table 5 Experimental results of KID index of the present application and the comparative method on 512*512*3 data set

[0107] Datasets AnimeFace ArtPainting Moongate Flat Fauvism StyleGAN2 242.83 113.75 93.14 214.47 220.14 DiffAug 148.51 18.42 48.04 247.41 117.10 ADA 16.02 12.26 43.21 62.89 86.64 APA 15.73 10.60 50.52 166.53 66.13 FastGAN 11.17 10.26 23.57 36.88 58.01 Ours 4.58 9.71 15.58 18.35 49.81

[0108] Table 6 Experimental results of FID index of the present application and the comparative method on 1024*1024*3 data set

[0109]

[0110]

[0111] Table 7 Experimental results of KID index of the present application and the comparative method on 1024*1024*3 data set

[0112] Datasets Pokemon Skulls Shells BrecaHAD MetFace StyleGAN2 33.36 209.22 161.98 176.32 55.53 DiffAug 54.73 38.12 50.68 31.62 11.55 ADA 29.12 12.33 - 21.38 10.23 APA 58.77 12.74 59.29 25.08 5.58 FastGAN 37.00 12.10 17.31 7.24 7.08 Ours 20.58 5.47 10.42 3.41 5.93

[0113] Tables 2, 3, 4, 5, 6, and 7 give the experimental results of the method of the present application and the comparative method on different resolution data sets, and it can be seen that in the limited sample quantity scene, the present application performs better in FID and KID indexes, proving that the present application can effectively improve the quality of generated pictures in the small sample scene, and realize more diverse and realistic small sample image generation.

[0114] In summary, the small sample image generation method fusing frequency domain information of the present application can significantly improve the diversity and authenticity of generated pictures in the small sample scene, and the quantitative and qualitative results verify the effectiveness and practicability of the system of the present application. At the same time, the pictures generated by the present application can be used for a wide range of tasks in the limited data scene, including data enhancement, classification, segmentation, etc. In addition, the present application fully considers the frequency domain information, which also provides a good reference for other deep learning related problems. As an important source of signal representation, frequency domain information can be extended to other related application scenarios, and the present application not only has strong application significance, but also has good reference and reference significance.

[0115] The above is a specific embodiment of the present application, and the present application is not limited to the examples. For those skilled in the art, the present application can be adapted to various different models, and can be adapted, adjusted and changed according to specific tasks. Any modification, replacement and improvement within the scope of the principles of the present application should be included in the scope of the claims of the present application.

Claims

1. A method for generating small sample images by fusing frequency domain information, used for generating real and diverse pictures in the scene of limited training data, characterized in that, The small sample image fusion frequency domain information generation method comprises: (1) a training picture preprocessing step, the preprocessed training picture is used for training the discriminator network and the generator network; The training picture includes a real picture from a training set and a generated picture from the generator network, the training picture is resized, randomly horizontally flipped, normalized to a mean of 0.5 and a variance of 0.5, and then converted into a tensor form to obtain the preprocessed training picture; (2) a step of training the discriminator network, the discriminator network is trained based on the preprocessed training picture, the discriminator network comprises a picture feature discrimination module, a high-frequency discrimination module and a discriminator network loss module; The picture feature discrimination module is used for feature extraction and discrimination of the preprocessed training picture, and discriminates whether the preprocessed training picture is the real picture or the generated picture; The high-frequency discrimination module is used for high-frequency information extraction and frequency domain discrimination of the preprocessed training picture, the high-frequency discrimination module uses Haar wavelet transform to extract the high-frequency information of the preprocessed training picture, the Haar wavelet transform includes two mirror image operations: wavelet pooling and wavelet unpooling operations, the wavelet pooling operation converts the preprocessed training picture into a frequency domain representation, and the wavelet unpooling operation reconstructs the frequency domain representation into a picture representation; The discriminator network loss module is used for calculating the constraint loss of the discriminator network according to the picture feature discrimination module and the high-frequency discrimination module, and the discriminator network is trained and iteratively updated according to the constraint loss of the discriminator network; (3) a step of training the generator network, the generator network is trained based on the discriminator network, the generator network comprises a mapping generated picture module, a frequency domain skip link module, a high-frequency information alignment module and a generator network loss module; The mapping generated picture module is used for mapping noise randomly sampled from a Gaussian distribution to obtain the generated picture; The frequency domain skip link module is used for extracting frequency domain information from the generated picture and transmitting the extracted frequency domain information to the deep layer of the generator network in a skip link manner; The high-frequency information alignment module is used for aligning the high-frequency information of the generated picture and the real picture; The generator network loss module is used for calculating the constraint loss of the generator network according to the discriminator network and the high-frequency information alignment module, and the generator network is trained and iteratively updated according to the constraint loss of the generator network; (4) alternately training the discriminator network and the generator network until the maximum number of iterations is reached.

2. The method of claim 1, wherein the frequency domain information is fused. The picture feature discrimination module is composed of a convolution network, a ReLU activation function, a Batch Normalization network and a down-sampling operation, and the preprocessed training picture is feature-extracted to obtain a feature representation of the preprocessed training picture.

3. The method of claim 1, wherein the frequency domain information is fused. The wavelet pooling operation of the Haar wavelet transform comprises four different kernels: LL T , LH T , HL T , HH T , wherein L and H represent low frequency and high frequency information filters respectively, the low frequency information filter L extracts the contour, outline coarse-grained information of the preprocessed training picture, and the high frequency information filter H extracts the edge, detailed texture fine-grained information of the preprocessed training picture.

4. The method of claim 1, wherein the frequency domain information is fused. The high-frequency discrimination module discriminates the high-frequency information to determine whether the high-frequency information is derived from the real picture or the generated picture after extracting the high-frequency information from the preprocessed training picture using the Haar wavelet transform, and the discrimination constraint is: wherein E denotes expectation, min() denotes minimization, HF r and HF f respectively denote the high frequency information of the real picture and the generated picture, I denotes the real picture, D H denotes a high frequency discriminative network of the discriminator network, z denotes noise randomly sampled in the Gaussian distribution, G(z) denotes the generated picture obtained by the mapping generated picture module of the generator network, and respectively denote discriminative constraints of the high frequency discriminative module for the generator network and the discriminator network.

5. The method of claim 1, wherein the frequency domain information is fused. The discriminator network loss module comprises a high-frequency discrimination loss, a discriminator network adversarial loss and a reconstruction loss, the high-frequency discrimination loss is a discrimination constraint of the discriminator network The discriminator network adversarial loss L D is: L D = -E x～I [min(0,-1+D(x))]-E x～G(z) [min(0,-1-D(x))] Wherein, E represents expectation, min() represents minimization, x represents the pre-processed training picture, D(x) is the picture feature discrimination module, z represents noise randomly sampled in the Gaussian distribution, G(z) represents the generated picture obtained by the mapping generated picture module of the generator network, I represents the real picture; the reconstruction loss L rec is: wherein f denotes the features that need to be reconstructed in the discriminator network, x denotes the pre-processed training picture, D encode () denotes the encoder network of the discriminator network, H(·) and T(·) denote small encoders for feature extraction of features and pictures, respectively, and I denotes the real picture; In summary, the discriminator network loss is generally represented as where L D represents the discriminator network adversarial loss, represents the high-frequency discriminative module discriminative constraint on the discriminator network, L rec represents the reconstruction loss.

6. The method of claim 1, wherein the method further comprises: The mapping generated picture module maps the noise randomly sampled from the Gaussian distribution into the generated picture through inverse convolution, upsampling, GLU, and jump link operations.

7. The method of claim 1, wherein the frequency domain information is fused. The frequency domain jump link module uses the Haar wavelet to extract frequency domain information from the generated picture and transmits the extracted frequency domain information to the deep layer of the generator network in a jump link manner: wherein F i+1 represents the features obtained by the (i+1)th layer of the generator network, represents the frequency domain information of the i-th layer of the generator network, Unpooling is the wavelet Unpooling operation, is the new feature obtained after using the frequency domain skip link for the (i+1)th layer.

8. The method of claim 1, wherein the frequency domain information is fused. The high-frequency information alignment module aligns the high-frequency information of the real picture and the generated picture, uses the high-frequency information of the real picture as self-supervised guiding information for the generator network to generate high-frequency information, and the high-frequency information alignment module constrains L align is: L align =||HF D -HF G ||1 wherein HF D denotes the high frequency information extracted from the real picture by the Haar wavelet, HF G denotes the high frequency information extracted from the generated picture by the Haar wavelet, ||·||1 denotes the L1 norm.

9. The method of claim 1, wherein the frequency domain information is fused. The generator network loss module comprises the generator network adversarial loss, the high-frequency discriminator module high-frequency discrimination constraint of the generator network and high-frequency information alignment loss; the generator network adversarial loss L G is: L G = -E z～N [D(G(z))] wherein z represents noise randomly sampled from the Gaussian distribution N, G is the mapping generated picture module, G(z) represents the generated picture obtained by the mapping generated picture module of the generator network, D is the picture feature discrimination module, and E represents expectation. In summary, the generator network loss is generally represented as where L align represents the high-frequency information alignment module constraint, represents the high-frequency discriminator module discriminative constraint on the generator network, L G represents the generator network adversarial loss.

Images