Image generation method and device based on importance weight and frequency domain modulation and medium

Abstract

The invention discloses an image generation method and device based on importance weight and frequency domain modulation and a medium, and relates to the technical field of image generation, and the method comprises the steps: obtaining a target style image and a target content image; inputting the target style image and the target content image into an image generation model to obtain a target generation image; the image generation model is a generator of a trained generative adversarial network, the trained generative adversarial network is obtained by training the generative adversarial network by using a training sample set based on an adversarial loss function containing an importance weight, and the generative adversarial network comprises a generator and a discriminator, the generator comprises a style encoder, a style encoder and a frequency domain modulation decoder, and the frequency domain modulation decoder is constructed based on wavelet transform and an AdaIN algorithm. The perception quality of the generated image is improved.

Description

Technical Field

[0001] The present application relates to the technical field of image generation, and in particular to an image generation method, device and medium based on importance weight and frequency domain modulation. Background Art

[0002] Compared to other forms of information, images are used extensively in our daily lives. Therefore, image processing research is of critical value and has a wide range of applicable scenarios.

[0003] Image generation can be divided into reference-based image generation and random sampling-based image generation. Reference-based image generation refers to sending a given target / style image into the style encoder network to obtain style features, and then sending the style features into the generator to generate an image with the target style. Random sampling-based image generation refers to randomly sampling from Gaussian noise to generate style features of the target domain, and then sending the style features into the generator to generate a new image of the target domain. This method can generate style images of the same target domain or images of different domains based on a Gaussian sampling noise.

[0004] With the complexity of application scenarios, image data is becoming more and more abundant. The image quality generated by traditional methods is poor, with many limitations, which is not conducive to practical application. The image generation method based on deep learning overcomes the defects of traditional methods, shows strong capabilities in data feature analysis and processing, and greatly improves the performance of image generation algorithms. At present, the image generation algorithm based on deep generation model has made great progress. It has been embedded in the application of various intelligent terminals as a key technology to support related functions. For example, image generation in the fields of short video production, advertising design, animation production, game rendering, etc. has a large number of practical application scenarios and huge commercial value. In addition, image generation technology also plays an important role in the field of content security. While the Internet brings convenience, it also makes it easy to spread forged images, videos and other content, which has a serious social impact. Therefore, forgery detection technology has become an important research content in the field of content security. And generation technology can provide forgery ideas and fake examples from the perspective of forgery, promoting the development of forgery detection technology.

[0005] However, most existing methods focus on studying image generation technology in the spatial domain, ignoring the impact of image frequency domain features on the perceived quality of generation, which results in the inability to generate high-frequency signals of images, affecting the further development of image generation quality. In addition, although with the development of big data, it has become relatively easy to obtain massive image data, there are still some image data that are difficult to obtain a large number of training images, such as medical images, ethnic art images, etc. Most of the current deep generation models require a large amount of training data to train the model. When the training data is small, these models are prone to overfitting, resulting in model collapse and failure to generate high-quality images. Therefore, exploring deep image generation models suitable for training with small amounts of data is also an urgent problem to be solved. Summary of the invention

[0006] The purpose of this application is to provide an image generation method, device and medium based on importance weight and frequency domain modulation to solve the problems of overfitting of the generation model under effective data training and low perceptual quality of the generated image.

[0007] To achieve the above objectives, this application provides the following solutions:

[0008] In a first aspect, the present application provides an image generation method based on importance weight and frequency domain modulation, comprising:

[0009] Obtain a target style image and a target content image;

[0010] The target style image and the target content image are both input into an image generation model to obtain a target generated image; the image generation model is a generator of a trained generative adversarial network, the trained generative adversarial network is based on an adversarial loss function containing importance weights, and is obtained by training the generative adversarial network using a training sample set, the generative adversarial network includes: a generator and a discriminator, the generator includes: a style encoder, a style encoder and a frequency domain modulation decoder, and the frequency domain modulation decoder is constructed based on wavelet transform and AdaIN algorithm.

[0011] Optionally, the training process of the image generation model includes:

[0012] Initialize the generative adversarial network;

[0013] Acquire the training sample set; the training sample set includes: a plurality of image groups, each image group includes: a sample style image and a sample content image;

[0014] Using the training sample set to alternately train the generator and the discriminator in the generative adversarial network to obtain a trained generative adversarial network;

[0015] The generator in the trained generative adversarial network is determined as the image generation model.

[0016] Optionally, alternately training a generator and a discriminator in a generative adversarial network using the training sample set to obtain a trained generative adversarial network includes:

[0017] The training times of any discriminator are determined as the current training times of the discriminator, and the training times of any generator are determined as the current training times of the generator;

[0018] Get the generator under the previous generator training times and the discriminator under the current discriminator training times; when the current discriminator training times is the first time, the generator under the previous generator training times is the initial generator;

[0019] Determine the generator under the previous generator training times as the generator under the current discriminator training times;

[0020] Input each image group into the generator under the current discriminator training times respectively, and obtain the sample generated image under the corresponding current discriminator training times;

[0021] Input each sample content image and the corresponding sample generated image at the current discriminator training times into the discriminator at the current discriminator training times, respectively, to obtain the discriminant features of each sample content image at the current discriminator training times and the discriminant features of each sample content image at the current discriminator training times;

[0022] Calculating a first importance weight at the current discriminator training times based on the discriminant features of the sample content image at the current discriminator training times using a first weight calculation formula;

[0023] Using the first final adversarial loss calculation formula, based on the discriminant features of each sample content image at the current discriminator training times, the discriminant features of each sample content image at the current discriminator training times, and the first importance weight at the current discriminator training times, the first final adversarial loss at the current discriminator training times is calculated; the first final adversarial loss is the adversarial loss of the discriminator with respect to the sample content image with the first importance weight introduced;

[0024] Using the first final adversarial loss under the current discriminator training times, the parameters of the discriminator under the current discriminator training times are updated to obtain the discriminator under the next discriminator training times;

[0025] Get the generator under the current generator training times;

[0026] Determine the discriminator for the next discriminator training times as the discriminator for the current generator training times;

[0027] Input each image group into the generator under the current generator training times respectively, and obtain the sample generated image under the current generator training times;

[0028] Input each sample content image and the corresponding sample generated image at the current generator training times into the generator at the current generator training times, respectively, to obtain the discriminant features of each sample content image at the current generator training times and the discriminant features of each sample content image at the current generator training times;

[0029] Calculating a second importance weight at the current generator training times based on the discriminant features of the sample content image at the current generator training times using a second weight calculation formula;

[0030] Using the second final adversarial loss calculation formula, based on the discriminant features of each sample content image at the current generator training times, the discriminant features of each sample content image at the current generator training times and the second importance weight at the current generator training times, the second final adversarial loss at the current generator training times is calculated; the second final adversarial loss is the adversarial loss of the generator with respect to the sample generated image with the introduction of the second importance weight;

[0031] Using the second final adversarial loss under the current generator training times, update the parameters of the generator under the current generator training times to obtain the generator under the next generator training times;

[0032] Determine whether the next generator training times reaches the preset training times;

[0033] If yes, the generative adversarial network composed of the discriminator under the current generator training times and the generator under the next generator training times is determined as the trained generative adversarial network;

[0034] If not, update the generator under the current discriminator training times to the generator under the current generator training times, update the discriminator under the current discriminator training times to the discriminator under the current generator training times, and return to "input each image group into the generator under the current discriminator training times respectively, and obtain the corresponding sample generated image under the current discriminator training times", until the preset training times are reached, and a trained generative adversarial network is obtained.

[0035] Optionally, the first weight calculation formula includes:

[0036]

[0037] Among them, β is the first importance weight; Exp(·) is the exponential function; D is the discriminator; x c,i is the sample content image in the i-th image group in the training sample set;c,i is the label of the sample content image in the i-th image group in the training sample set; batch is the number of image groups in the training sample set.

[0038] Optionally, the first final confrontation loss calculation formula includes:

[0039] L' D =E[1-D(x g ,y s ))+βE(D(x c ,y c )];

[0040] Among them, L' D is the first final adversarial loss; E[·] is the mean function; x g Generate an image for the sample, x g =G(c,s), G is the generator, c is the sample content feature matrix, s is the sample style feature vector; y s is the label of the sample style image; x c is the sample content image; y c is the label of the sample content image.

[0041] Optionally, the second weight calculation formula includes:

[0042]

[0043] Among them, α is the second importance weight; x g,i Generate an image for the sample corresponding to the i-th image group in the training sample set; y s,i is the label of the sample style image in the i-th image group in the training sample set.

[0044] Optionally, the second final adversarial loss calculation formula includes:

[0045] L' G =αE[D(x g ,y s ))];

[0046] Among them, L' G For the second final fight loss.

[0047] Optionally, inputting both the target style image and the target content image into an image generation model to obtain a target generated image includes:

[0048] Inputting the target style image into a style encoder in an image generation model to obtain a target style feature vector;

[0049] Inputting the target content image into a content encoder in an image generation model to obtain a target content feature matrix;

[0050] The target style feature vector and the target content feature matrix are input into a frequency domain modulation decoder in an image generation model to obtain the target generated image.

[0051] In a second aspect, the present application provides a computer device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement any of the above-described image generation methods based on importance weight and frequency domain modulation.

[0052] In a third aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-mentioned image generation methods based on importance weight and frequency domain modulation.

[0053] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0054] The present application discloses an image generation method, device and medium based on importance weight and frequency domain modulation, and obtains a target style image and a target content image; inputs both the target style image and the target content image into an image generation model to obtain a target generated image; the image generation model is a generator of a trained generative adversarial network, and the trained generative adversarial network is obtained by training the generative adversarial network based on an adversarial loss function containing importance weights using a training sample set, and the generative adversarial network includes: a generator and a discriminator, and the generator includes: a style encoder, a style encoder and a frequency domain modulation decoder, and the frequency domain modulation decoder is constructed based on wavelet transform and AdaIN algorithm. The present application designs an adversarial loss function based on importance weights, strengthens the weight of high-quality generated images during generator training, and allows the discriminator to assign more confidence to high-quality generated images during the discrimination process, thereby reducing the problem of generator adversarial loss divergence during training, and improving the ability of the generator to deceive the discriminator during training; the problem that the discriminator tends to distinguish true from false images based on high-frequency signals during training. From the perspective of enhancing the high-frequency signals of the generated image, a frequency domain modulation decoder is constructed based on wavelet transform and AdaIN algorithm. The wavelet high-frequency features of the style image and content image are sent to the frequency domain modulation decoder through the mean and variance statistical characteristics, which enables the network to learn the high-frequency characteristics of the style image and content image, thereby improving the high-frequency perceptual quality of the generated image. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0056] Figure 1 A schematic diagram of a flow chart of an image generation method based on importance weight and frequency domain modulation provided in one embodiment of the present application;

[0057] Figure 2 Schematic diagram of the training process of the image generation model;

[0058] Figure 3 Schematic diagram of the process of adversarial loss adjustment for importance weights;

[0059] Figure 4 It is a schematic diagram of the network structure of the frequency domain modulation decoder;

[0060] Figure 5 Schematic diagram of Wavelet-AdaIN structure;

[0061] Figure 6 is the original image during wavelet transformation;

[0062] Figure 7 It is a schematic diagram of the low-frequency component of the original image after wavelet transformation;

[0063] Figure 8 It is the high-frequency component image in the vertical direction after the original image is transformed by wavelet;

[0064] Fig. 9 It is the high-frequency component image in the horizontal direction after the original image is transformed by wavelet;

[0065] Fig.10 It is the high-frequency component image in the diagonal direction after the original image is transformed by wavelet;

[0066] Fig.11 It is a schematic diagram of the total high-frequency components of the original image after wavelet transformation;

[0067] Fig.12 A schematic diagram of the structure of a computer device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0068] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0069] The purpose of this application is to provide an image generation method, device and medium based on importance weight and frequency domain modulation, aiming to improve the perceptual quality of the generated image.

[0070] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0071] In an exemplary embodiment, Figure 1 As shown, the image generation method based on importance weight and frequency domain modulation in this embodiment includes:

[0072] Step 101: Obtain a target style image and a target content image.

[0073] Step 102: Input the target style image and the target content image into the image generation model to obtain the target generated image; the image generation model is a generator of a trained generative adversarial network, the trained generative adversarial network is based on an adversarial loss function containing importance weights, and is obtained by training the generative adversarial network using a training sample set, the generative adversarial network includes: a generator and a discriminator, the generator includes: a style encoder, a style encoder and a frequency domain modulation decoder, and the frequency domain modulation decoder is constructed based on wavelet transform and AdaIN algorithm.

[0074] As an optional implementation, Figure 2 As shown in Figure 1, the training process of the image generation model includes:

[0075] Step 1: Initialize the generative adversarial network.

[0076] Step 2: Obtain a training sample set; the training sample set includes: multiple groups of image groups, each image group includes: a sample style image and a sample content image.

[0077] Step 3: Use the training sample set to alternately train the generator and discriminator in the generative adversarial network to obtain a trained generative adversarial network.

[0078] Step 4: Determine the generator in the trained generative adversarial network as the image generation model.

[0079] As an optional implementation, step 3: includes:

[0080] Step 301: The number of training times of any discriminator is determined as the number of training times of the current discriminator, and the number of training times of any generator is determined as the number of training times of the current generator.

[0081] Step 302: Obtain the generator under the previous generator training times and the discriminator under the current discriminator training times; when the current discriminator training times is the first time, the generator under the previous generator training times is the initial generator.

[0082] Step 303: Determine the generator under the previous generator training times as the generator under the current discriminator training times.

[0083] Step 304: Input each image group into the generator under the current discriminator training times respectively, and obtain the sample generated image under the corresponding current discriminator training times.

[0084] Step 305: Input each sample content image and the corresponding sample generated image at the current discriminator training times into the discriminator at the current discriminator training times respectively, and obtain the discriminant features of each sample content image at the current discriminator training times and the discriminant features of each sample content image at the current discriminator training times.

[0085] Step 306: using the first weight calculation formula, based on the discriminant features of the sample content image at the current discriminator training times, calculate the first importance weight at the current discriminator training times.

[0086] As an optional implementation, the first weight calculation formula includes:

[0087]

[0088] Among them, β is the first importance weight; Exp(·) is the exponential function; D is the discriminator; x c,i is the sample content image in the i-th image group in the training sample set; c,i is the label of the sample content image in the i-th image group in the training sample set; batch is the number of image groups in the training sample set.

[0089] Step 307: Calculate the first final adversarial loss at the current discriminator training times based on the discriminant features of each sample content image at the current discriminator training times, the discriminant features of each sample content image at the current discriminator training times, and the first importance weight at the current discriminator training times using the first final adversarial loss calculation formula; the first final adversarial loss is the adversarial loss of the discriminator with respect to the sample content image with the introduction of the first importance weight.

[0090] As an optional implementation, the first final adversarial loss calculation formula includes:

[0091] L' D =E[1-D(x g ,y s ))+βE(D(x c ,y c )].

[0092] Among them, L' D is the first final adversarial loss; E[·] is the mean function; x g Generate an image for the sample, x g =G(c,s), G is the generator, c is the sample content feature matrix, s is the sample style feature vector; y s is the label of the sample style image; x c is the sample content image; y c is the label of the sample content image.

[0093] Step 308: Use the first final adversarial loss under the current discriminator training times to update the parameters of the discriminator under the current discriminator training times to obtain the discriminator under the next discriminator training times.

[0094] Step 309: Get the generator under the current generator training times.

[0095] Step 310: Determine the discriminator at the next discriminator training times as the discriminator at the current generator training times.

[0096] Step 311: Input each image group into the generator under the current generator training times respectively to obtain the sample generated image under the current generator training times.

[0097] Step 312: Input each sample content image and the corresponding sample generated image at the current generator training times into the generator at the current generator training times respectively, and obtain the discriminant features of each sample content image at the current generator training times and the discriminant features of each sample content image at the current generator training times.

[0098] Step 313: Using the second weight calculation formula, calculate the second importance weight at the current generator training times based on the discriminant features of the sample content image at the current generator training times.

[0099] As an optional implementation, the second weight calculation formula includes:

[0100]

[0101] Among them, α is the second importance weight; x g,i Generate an image for the sample corresponding to the i-th image group in the training sample set; ys,i is the label of the sample style image in the i-th image group in the training sample set.

[0102] Step 314: Calculate the second final adversarial loss at the current generator training times based on the discriminant features of each sample content image at the current generator training times, the discriminant features of each sample content image at the current generator training times and the second importance weight at the current generator training times using the second final adversarial loss calculation formula; the second final adversarial loss is the adversarial loss of the generator with respect to the sample generated image with the introduction of the second importance weight.

[0103] As an optional implementation, the second final adversarial loss calculation formula includes:

[0104] L' G =αE[D(x g ,y s ))].

[0105] Among them, L' G For the second final fight loss.

[0106] Step 315: Use the second final adversarial loss under the current generator training times to update the parameters of the generator under the current generator training times to obtain the generator under the next generator training times.

[0107] Step 316: Determine whether the next generator training times reaches the preset training times.

[0108] Step 317: If so, the generative adversarial network composed of the discriminator under the current generator training times and the generator under the next generator training times is determined as the trained generative adversarial network.

[0109] Step 318: If not, update the generator under the current discriminator training times to the generator under the current generator training times, update the discriminator under the current discriminator training times to the discriminator under the current generator training times, and return to step 304 until the preset training times are reached to obtain a trained generative adversarial network.

[0110] Specifically, in the study of image generation with limited data, the main problem is that the discriminator is prone to overfitting, which further leads to the divergence of the generator loss during training. Therefore, this application combines learnable importance weights (i.e., first importance weights and second importance weights) with adversarial loss to solve this problem. The purpose of introducing importance weights is to track the discriminator's predictions about real images (including sample style images and sample content images) and sample generated images, so that high-quality sample generated images have a greater impact on the overall during the training of the generator. The process of the adversarial loss adjustment method of importance weights is as follows: Figure 3 shown.

[0111] Generative adversarial networks (GANs) use adversarial loss to ensure the game between the generator and the discriminator. The generator aims to obtain high-quality generated images to deceive the discriminator, and the discriminator aims to distinguish between real generated images and fake generated images. The adversarial loss function in the general GANs training process is:

[0112]

[0113] in, is the adversarial loss function value of general GANs.

[0114] The adversarial loss function in the general GANs training process is decomposed to obtain the adversarial loss of the training discriminator in the general GANs training process and the adversarial loss of the training generator in the general GANs training process:

[0115] L D =E[1-D(x g ,y s ))+E(D(x c ,y c )].

[0116] L G =E[D(x g ,y s ))].

[0117] Among them, L D is the adversarial loss for training the discriminator; L G is the adversarial loss for training the generator.

[0118] In image generation tasks with limited data, the discriminator is extremely prone to overfitting. In order to solve the problem of overfitting, importance weights are introduced on the basis of adversarial loss to strengthen the influence of high-quality generated images and improve the balance between the generator and the discriminator game. That is, the first importance weight is added to the adversarial loss of the training discriminator in the general GANs training process to obtain the first adversarial loss calculation formula, and the second importance weight is added to the adversarial loss of the training generator in the general GANs training process to obtain the second adversarial loss calculation formula.

[0119] During the generator training process, a higher first importance weight is given to high-quality generated images, thereby reducing the generator's loss, preventing the adversarial loss divergence problem during training, and further improving the generator's ability to deceive the discriminator during backpropagation. During the generator training process, the second importance weight can strengthen the influence of high-quality generated images, thereby improving the generator's ability to deceive the discriminator.

[0120] As an optional implementation, step 102 includes:

[0121] Step 1021: Input the target style image into the style encoder in the image generation model to obtain a target style feature vector.

[0122] Step 1022: Input the target content image into the content encoder in the image generation model to obtain a target content feature matrix.

[0123] Step 1023: Input the target style feature vector and the target content feature matrix into the frequency domain modulation decoder in the image generation model to obtain the target generated image.

[0124] Specifically, GANs give priority to generating low-frequency signals during training and tend to ignore the generation of high-frequency signals. Therefore, in order to improve the frequency perception ability of the image generation model, this application designs a high-frequency feature fusion module Wavelet-AdaIN based on wavelet feature guidance, which aims to learn the high-frequency statistical features of the real data distribution to improve the perceptual quality of the generated image. This module is embedded in the frequency domain modulation decoder. The network structure of the frequency domain modulation decoder is as follows: Figure 4 As shown, the Wavelet-AdaIN design structure is as follows Figure 5 shown.

[0125] The high-frequency feature fusion module supplements the fusion of high-frequency information on the basis of AdaIN, that is, the high-frequency information from the style image is adaptively fused with the high-frequency features in the frequency domain modulation decoder, and then the fused features are modulated to modulate the output of each layer in the frequency domain modulation decoder through AdaIN. In this way, the final generated image integrates the high-frequency features from the real image (including style images and content images), improves the detail quality of the generated image, and does not hinder the convergence of the network during training. The sample style image and the target style image are collectively referred to as style images, and the sample content image and the target content image are collectively referred to as content images.

[0126] The specific implementation process of the high-frequency feature fusion module includes:

[0127] 1. Haar wavelet filter is used as the basis of wavelet transform, which includes two processes: wavelet transform and inverse wavelet transform. Wavelet transform converts the image from spatial domain to frequency domain, and inverse wavelet transform reconstructs the frequency domain components into spatial domain. Haar wavelet transform includes four kernels: LL T , LH T , HL T and HH T .

[0128]

[0129] Among them, T represents transpose; L and H represent low-pass filter and high-pass filter respectively. Low-pass filter focuses on low-frequency signals, such as contours and structural information. In contrast, high-pass filter emphasizes high-frequency signals and captures fine-grained details, such as subtle edges. Figure 6-Figure 11 The original image and the frequency components of the original image obtained by Haar wavelet transform are shown.

[0130] Wavelet transform decomposes the features into LL T , LH T , HL T and HH T Four frequency components. Due to LH T and HL T Contains some low-frequency components, resulting in some low-frequency information of the content image being sent to the frequency domain modulation decoder, which affects the network's learning of style information and is not conducive to network convergence. Therefore, only HH is selected in the Wavelet-AdaIN module. T The components are used to calculate the mean and variance, and then the obtained mean and variance are sent to the frequency domain modulation decoder for feature fusion.

[0131] 2. There are three inputs in the Wavelet-AdaIN module: style features E from the style encoder s , the deconvolution layer output feature F in the frequency domain modulation decoder i Let C be the number of channels of the feature map, H1 be the height of the feature map, W be the width of the feature map, and N be the number of images in the current training batch.

[0132] The execution of the Wavelet-AdaIN module can be divided into the following two steps (n∈N,c∈C,y∈H1,x∈W):

[0133] (1) First, perform fusion learning of high-frequency features to obtain fine high-frequency detail features. The expression in the learning process is:

[0134]

[0135] Among them, F H is the high-frequency detail feature; n,y,x is the adaptive summation of the high-frequency feature mean from the frequency domain modulation decoder and the high-frequency feature mean of the style encoder, through a parameter weight α adaptively learned during network training h To control their respective proportions; μ n,y,x Output feature F for the deconvolution layer i The high-frequency mean and variance of n,y,x Output feature F for the deconvolution layer iThe high-frequency variance of n,y,x It is the adaptive summation of the high-frequency feature variance from the frequency domain modulation decoder and the high-frequency feature variance of the encoder, through a parameter weight β adaptively learned during network training h To control their respective proportions.

[0136] γ n,y,x and β n,y,x is learnable, γ n,y,x and β n,y,x They can be expressed as:

[0137]

[0138] in, is the style feature E from the style encoder s The high frequency mean of . is the style feature E from the style encoder s The high-frequency variance of style feature E s And the deconvolution layer output feature F i The size of the feature map is consistent, that is, the style feature E s And the deconvolution layer output feature F i The feature maps have the same number of images N, number of feature channels C, height H1 and width W.

[0139] μ n,y,x and σ n,y,x They can be expressed as:

[0140]

[0141] and They can be expressed as:

[0142]

[0143] (2) Secondly, after obtaining F H After that, AdaIN is executed to learn the style features, which can be expressed as:

[0144]

[0145] Where F is the output feature after the Wavelet-AdaIN module; γ(s) is the mean of the style feature vector s (referred to as style vector s); μ is F H The mean value on the channel; σ is F H Variance on the channel; β(S) is the variance of the style feature vector s.

[0146] Ultimately, this module can fuse high-frequency detail information from real images in the process of generating images, enhance the detail features of the generated images, and thus gradually output higher-quality generated images in the frequency domain modulation decoder.

[0147] Advantages of this application:

[0148] (1) The problem of adversarial loss divergence during model training under limited data is caused by the extremely low confidence of the discriminator in the generated images during training. Therefore, this application designs an adversarial loss based on importance weights, which mainly strengthens the weight of high-quality generated images during generator training, allowing the discriminator to assign more confidence to high-quality generated images during the discrimination process, thereby reducing the problem of adversarial loss divergence during training and improving the ability of the generator to deceive the discriminator during training.

[0149] (2) The problem that the discriminator tends to distinguish true and false images based on high-frequency signals during training. From the perspective of enhancing the high-frequency signals of the generated image, the applicants proposed a high-frequency feature fusion module guided by wavelet features. This module is mainly the normalization of Wavelet-AdaIN. This module sends the wavelet high-frequency features of the real image into the frequency domain modulation decoder through the statistical characteristics of mean and variance, enabling the network to learn the high-frequency characteristics of the real image, thereby improving the high-frequency perception quality of the generated image.

[0150] In an exemplary embodiment, a computer device is provided, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement an image generation method based on importance weight and frequency domain modulation.

[0151] In an exemplary embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, an image generation method based on importance weight and frequency domain modulation is implemented.

[0152] In an exemplary embodiment, a computer program product is provided, including a computer program, which implements an image generation method based on importance weight and frequency domain modulation when executed by a processor.

[0153] In an exemplary embodiment, a computer device is provided. The computer device may be a server or a terminal. The internal structure diagram thereof may be as follows: Fig.12As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, an image generation method based on importance weight and frequency domain modulation is implemented.

[0154] Those skilled in the art will understand that Fig.12 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0155] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0156] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0157] The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. The non-relational database may include a distributed database based on blockchain, etc., but is not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but is not limited thereto.

[0158] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0159] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. An image generation method based on importance weight and frequency domain modulation, characterized in that: The image generation method based on importance weight and frequency domain modulation includes: Obtain a target style image and a target content image; The target style image and the target content image are both input into an image generation model to obtain a target generated image; the image generation model is a generator of a trained generative adversarial network, the trained generative adversarial network is based on an adversarial loss function containing importance weights, and is obtained by training the generative adversarial network using a training sample set, the generative adversarial network includes: a generator and a discriminator, the generator includes: a style encoder, a style encoder and a frequency domain modulation decoder, and the frequency domain modulation decoder is constructed based on wavelet transform and AdaIN algorithm.

2. The image generation method based on importance weight and frequency domain modulation according to claim 1, characterized in that: The training process of the image generation model includes: Initialize the generative adversarial network; Acquire the training sample set; the training sample set includes: a plurality of image groups, each image group includes: a sample style image and a sample content image; Using the training sample set to alternately train the generator and the discriminator in the generative adversarial network to obtain a trained generative adversarial network; The generator in the trained generative adversarial network is determined as the image generation model.

3. The image generation method based on importance weight and frequency domain modulation according to claim 2, characterized in that: The generator and the discriminator in the generative adversarial network are alternately trained using the training sample set to obtain a trained generative adversarial network, including: The training times of any discriminator are determined as the current training times of the discriminator, and the training times of any generator are determined as the current training times of the generator; Get the generator under the previous generator training times and the discriminator under the current discriminator training times; when the current discriminator training times is the first time, the generator under the previous generator training times is the initial generator; Determine the generator under the previous generator training times as the generator under the current discriminator training times; Input each image group into the generator under the current discriminator training times respectively, and obtain the sample generated image under the corresponding current discriminator training times; Input each sample content image and the corresponding sample generated image at the current discriminator training times into the discriminator at the current discriminator training times, respectively, to obtain the discriminant features of each sample content image at the current discriminator training times and the discriminant features of each sample content image at the current discriminator training times; Calculating a first importance weight at the current discriminator training times based on the discriminant features of the sample content image at the current discriminator training times using a first weight calculation formula; Using the first final adversarial loss calculation formula, based on the discriminant features of each sample content image at the current discriminator training times, the discriminant features of each sample content image at the current discriminator training times, and the first importance weight at the current discriminator training times, the first final adversarial loss at the current discriminator training times is calculated; the first final adversarial loss is the adversarial loss of the discriminator with respect to the sample content image with the first importance weight introduced; Using the first final adversarial loss under the current discriminator training times, the parameters of the discriminator under the current discriminator training times are updated to obtain the discriminator under the next discriminator training times; Get the generator under the current generator training times; Determine the discriminator for the next discriminator training times as the discriminator for the current generator training times; Input each image group into the generator under the current generator training times respectively, and obtain the sample generated image under the current generator training times; Input each sample content image and the corresponding sample generated image at the current generator training times into the generator at the current generator training times, respectively, to obtain the discriminant features of each sample content image at the current generator training times and the discriminant features of each sample content image at the current generator training times; Calculating a second importance weight at the current generator training times based on the discriminant features of the sample content image at the current generator training times using a second weight calculation formula; Using the second final adversarial loss calculation formula, based on the discriminant features of each sample content image at the current generator training times, the discriminant features of each sample content image at the current generator training times and the second importance weight at the current generator training times, the second final adversarial loss at the current generator training times is calculated; the second final adversarial loss is the adversarial loss of the generator with respect to the sample generated image with the introduction of the second importance weight; Using the second final adversarial loss under the current generator training times, update the parameters of the generator under the current generator training times to obtain the generator under the next generator training times; Determine whether the next generator training times reaches the preset training times; If yes, the generative adversarial network composed of the discriminator under the current generator training times and the generator under the next generator training times is determined as the trained generative adversarial network; If not, update the generator under the current discriminator training times to the generator under the current generator training times, update the discriminator under the current discriminator training times to the discriminator under the current generator training times, and return to "input each image group into the generator under the current discriminator training times, and obtain the corresponding sample generated image under the current discriminator training times" until the preset training times are reached to obtain a trained generative adversarial network.

4. The image generation method based on importance weight and frequency domain modulation according to claim 3, characterized in that: The first weight calculation formula includes: Among them, β is the first importance weight; Exp(·) is the exponential function; D is the discriminator; x c,i is the sample content image in the i-th image group in the training sample set; c,i is the label of the sample content image in the i-th image group in the training sample set; batch is the number of image groups in the training sample set.

5. The image generation method based on importance weight and frequency domain modulation according to claim 4, characterized in that: The first final confrontation loss calculation formula includes: L' D =E[1-D(x g ,y s ))+βE(D(x c ,y c )]; Among them, L' D is the first final adversarial loss; E[·] is the mean function; x g Generate an image for the sample, x g =G(c,s), G is the generator, c is the sample content feature matrix, s is the sample style feature vector; y s is the label of the sample style image; x c is the sample content image; y c is the label of the sample content image.

6. The image generation method based on importance weight and frequency domain modulation according to claim 5, characterized in that: The second weight calculation formula includes: Among them, α is the second importance weight; x g,i Generate an image for the sample corresponding to the i-th image group in the training sample set; y s,i is the label of the sample style image in the i-th image group in the training sample set.

7. The image generation method based on importance weight and frequency domain modulation according to claim 6, characterized in that: The second final confrontation loss calculation formula includes: L' G =αE[D(x g ,y s ))]; Among them, L' G For the second final fight loss.

8. The image generation method based on importance weight and frequency domain modulation according to claim 1, characterized in that: Inputting the target style image and the target content image into an image generation model to obtain a target generated image includes: Inputting the target style image into a style encoder in an image generation model to obtain a target style feature vector; Inputting the target content image into a content encoder in an image generation model to obtain a target content feature matrix; The target style feature vector and the target content feature matrix are input into a frequency domain modulation decoder in an image generation model to obtain the target generated image.

9. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the image generation method based on importance weight and frequency domain modulation as described in any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the image generation method based on importance weight and frequency domain modulation described in any one of claims 1 to 8 is implemented.

Images