Generative adversarial network training method, image style transfer method and device

Abstract

The embodiment of the present invention discloses a generative adversarial network training method, an image style transfer method and a device. The training method comprises: obtaining a content image and a style image of the same style, initializing network parameters of a generator and a discriminator of the generative adversarial network, and a style momentum accumulation of the generator, extracting the style image and inputting it into a style feature extraction network to obtain a style feature to update the style momentum accumulation, extracting the content image and inputting it into a generator to inject the style momentum accumulation into the content image to obtain an output image after style transfer, inputting the content image and the output image into a discriminator for training to obtain a judgment value of the output image, and adjusting the network parameters of the generator and the discriminator. On the one hand, the decoupling of content and style is achieved, and on the other hand, there is no need to obtain a large number of stylized images of content images for training as labels for supervised training, so that both training data can be easily obtained and ideal stylized images can be obtained.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of image processing technology, and in particular to a generative adversarial network training method, an image style transfer method and a device. Background Art

[0002] Comicization is an important technology in the field of computer vision and is widely used in content production, filmmaking, and entertainment videos. Comicization means that given a content image, the content image is transformed into a certain comic style image. The comic style image retains the content, posture and other information of the content image.

[0003] At present, image cartooning can be achieved through 3D rendering or by training neural networks. 3D rendering is used to make cartoon characters in games or videos, or to generate cartoon images using image filters and mathematical optimization methods. However, 3D rendering methods are difficult to produce cartoon images with a strong sense of art, and it is also difficult to produce high-level abstraction like artists, such as fresh line abstraction. For neural networks, content images and style images are required as training inputs when training neural networks, and the cartoonized images of content images are used as labels for supervised learning. On the one hand, the content images and style images are coupled and trained in a network, and the style and content cannot be flexibly constrained, resulting in the inability of the cartoonized images to maintain the content of the content images and achieve a real cartooning effect. On the other hand, it is difficult to obtain a large number of cartoonized images of content images as labels.

[0004] In summary, existing image cartoonization methods have the problems of poor cartoonization effect and difficulty in obtaining training data. Summary of the invention

[0005] The embodiments of the present invention provide a generative adversarial network training method, an image style transfer method, an apparatus, an electronic device and a storage medium to solve the problems of poor cartooning effect and difficulty in obtaining training data in existing image cartooning methods.

[0006] In a first aspect, an embodiment of the present invention provides a method for training a generative adversarial network, comprising:

[0007] Get multiple content images and multiple style images of the same style;

[0008] Initialize the network parameters of the generator and discriminator of the generative adversarial network, and initialize the style momentum accumulation of the generator;

[0009] Extracting the style image and inputting it into a pre-trained style feature extraction network to obtain style features;

[0010] updating the style momentum accumulation amount according to the style feature;

[0011] Extracting the content image and inputting it into the generator, so as to inject the style momentum accumulation into the content image through the generator to obtain an output image after style transfer;

[0012] Inputting the content image and the output image into the discriminator for training to obtain a judgment value of the output image;

[0013] The network parameters of the generator and the discriminator are adjusted according to the decision value, the output image, the content image and the style image.

[0014] In a second aspect, an embodiment of the present invention provides an image style transfer method, comprising:

[0015] Get the image to be processed;

[0016] Determining a generator in response to the image style selected by the user, the generator being used to transfer the image style selected by the user to the image to be processed;

[0017] Inputting the image to be processed into the generator to obtain an image after style transfer;

[0018] Wherein, the generator is trained by the generative adversarial network training method described in any embodiment of the present invention.

[0019] In a third aspect, an embodiment of the present invention provides a generative adversarial network training device, comprising:

[0020] A training data acquisition module, used to acquire multiple content images and multiple style images of the same style;

[0021] An initialization module, used to initialize the network parameters of the generator and the discriminator of the generative adversarial network, and to initialize the style momentum accumulation of the generator;

[0022] A style feature extraction module, used to extract the style features of the style image and input it into a pre-trained style feature extraction network;

[0023] A style momentum accumulation updating module, used for updating the style momentum accumulation according to the style feature;

[0024] A generator training module, used for extracting the content image and inputting it into the generator, so as to inject the style momentum accumulation into the content image through the generator to obtain an output image after style transfer;

[0025] A discriminator training module, used for inputting the content image and the output image into the discriminator for training to obtain a judgment value;

[0026] A network parameter adjustment module is used to adjust the network parameters of the generator and the discriminator according to the determination value, the output image, the content image and the style image.

[0027] In a fourth aspect, an embodiment of the present invention provides an image style transfer device, including:

[0028] The original image and target image acquisition module is used to acquire the original image containing the first face and the target image containing the second face;

[0029] An image face-swapping module, used for inputting the original image and the target image into a generator of a generative adversarial network, and obtaining a face-swapping image of the original image after replacing the first face with the second face;

[0030] Wherein, the generator is trained by the generative adversarial network training method described in any embodiment of the present invention.

[0031] In a fifth aspect, an embodiment of the present invention provides an electronic device, the electronic device comprising:

[0032] one or more processors;

[0033] a storage device for storing one or more programs,

[0034] When the one or more programs are executed by the one or more processors, the one or more processors implement the generative adversarial network training method and / or the image style transfer method described in any embodiment of the present invention.

[0035] In a sixth aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the generative adversarial network training method and / or the image style transfer method described in any embodiment of the present invention.

[0036] In an embodiment of the present invention, a generator is provided with a style momentum accumulation amount, and the style momentum accumulation amount is updated by extracting a style image of the same style and inputting it into a trained style feature extraction network. The generator only needs to input a content image to obtain an output image after style transfer, and then adjusts the network parameters of the generator and the discriminator according to the judgment value of the discriminator on the output image, the output image, the content image and the style image. After the training is completed, the final style momentum accumulation amount can contain the style information of the style. On the one hand, the style image is input into a trained style feature network to update the style momentum accumulation amount of the generator, and the content image is input into the generator to train the generator, thereby realizing the decoupling of content and style. The generator is more focused on injecting the style momentum accumulation amount into the content image to learn the style. It is easier to constrain the content and style in the generator, so that the image after style transfer can not only maintain the content of the content image, but also learn the style of the style image, thereby improving the stylization effect. On the other hand, there is no need to obtain a large number of stylized images of the content image as labels for supervised training. During training, it is sufficient to obtain the content image and multiple arbitrary style images of the same style, thereby reducing the difficulty of obtaining training data and making it easy to obtain training data. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 is a flowchart of the steps of a generative adversarial network training method provided in Embodiment 1 of the present invention;

[0038] Figure 2A is a flowchart of a method for training a generative adversarial network provided in Embodiment 2 of the present invention;

[0039] Figure 2B is a schematic diagram of each network of an embodiment of the present invention;

[0040] Figure 3 is a flowchart of a method for image style transfer provided by Embodiment 3 of the present invention;

[0041] Figure 4 It is a structural block diagram of a generative adversarial network training device provided by Embodiment 4 of the present invention;

[0042] Figure 5 is a structural block diagram of an image style transfer device provided by Embodiment 5 of the present invention;

[0043] Figure 6 It is a structural schematic diagram of an electronic device provided in Embodiment 6 of the present invention. DETAILED DESCRIPTION

[0044] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only the parts related to the present invention, rather than all structures, are shown in the accompanying drawings. In the absence of conflict, the embodiments in this application and the features in the embodiments may be combined with each other.

[0045] Embodiment 1

[0046] Figure 1 A flow chart of the steps of a generative adversarial network training method provided in the first embodiment of the present invention. The embodiment of the present invention can be applied to training a generative adversarial network to perform style transfer on an image through a generator of a trained generative adversarial network. The method can be performed by a generative adversarial network training device of the embodiment of the present invention. The generative adversarial network training device can be implemented by hardware or software and integrated in the electronic device provided in the embodiment of the present invention. Specifically, Figure 1 As shown, the generative adversarial network training method of the embodiment of the present invention may include the following steps:

[0047] S101: Acquire multiple content images and multiple style images of the same style.

[0048] The content image can be an image of the style of the image to be edited, and the style image can be an image of a specified style, wherein the style of the image can be a comic style, a martial arts style, an artistic style, etc., and each style can be further subdivided. When obtaining training data, any number of images can be obtained as content images, and then any number of images of the same style can be obtained as style images. The purpose of training a generative adversarial network in an embodiment of the present invention is to transfer the style of the style image to the content image through a generator to obtain a style-transferred image, so that the style-transferred image can maintain the content of the content image and have the style of the style image.

[0049] S102, initializing the network parameters of the generator and the discriminator of the generative adversarial network, and initializing the style momentum accumulation of the generator.

[0050] In an embodiment of the present invention, a Generative Adversarial Network (GANs) includes a generator and a discriminator. The generator is used to generate a new image after inputting data, and the discriminator is used to determine the probability of whether the new image is real. In an embodiment of the present invention, the generator and the discriminator can be neural networks, and the network parameters of the generator and the discriminator of the generative adversarial network can be initialized. In addition, the generator can also be provided with a style momentum accumulation, which contains style information of a specified style. The generator is used to inject the style momentum accumulation into the content image to obtain an image after the content image style is transferred.

[0051] In an optional embodiment of the present invention, the generator may include an encoding network, a decoding network and a residual network, wherein the residual network may be a pre-trained network, and the initialization generator may be the network parameters of the encoding network and the decoding network of the initialization generator, as well as the style momentum accumulator of the initialization generator.

[0052] S103: extract the style image and input it into a pre-trained style feature extraction network to obtain style features.

[0053] Specifically, a style feature extraction network can be pre-trained, and the style feature extraction network can extract style features of any style image. In each round of iterative training of the generator and the discriminator, a style image can be randomly extracted from multiple style images of the same style, and the style image can be input into the style feature extraction network to extract style features.

[0054] S104: Update the style momentum accumulation according to the style feature.

[0055] In one example of the present invention, the style feature extraction network may include a downsampling convolutional neural network, a global pooling layer and a fully connected layer. The style image obtains a global feature after passing through the downsampling convolutional neural network and the global pooling layer. The global feature obtains a style latent variable after passing through the fully connected layer. The style latent variable contains the style information of the style image. The style momentum accumulation amount initialized by the generator can be updated through the style latent variable.

[0056] S105: extract the content image and input it into the generator, so that the style momentum accumulation amount is injected into the content image through the generator to obtain an output image after style transfer.

[0057] In an embodiment of the present invention, a generator may include an encoding network, a decoding network and a residual network. In the generator, a content feature map may be extracted from a content image through the encoding network, and a style momentum accumulator may be injected into the content feature map using a residual network to generate a style transfer feature map, and the style transfer feature map may be decoded in a decoding network to obtain a style-transferred output image, wherein the content feature map may be a feature map expressing attributes such as a posture and structure of an object in the content image. For example, if the content image contains a face, the content feature map may be a feature map expressing attributes such as a posture, a face structure (such as key points), and a face expression in the content image.

[0058] S106: Input the content image and the output image into the discriminator for training to obtain a judgment value of the output image.

[0059] In the training process of the adversarial generative network, the purpose of the generator is to generate realistic images as much as possible to deceive the discriminator, and the purpose of the discriminator is to distinguish the images generated by the generator from the real images. By alternately training the generator and the discriminator, the training of the generator and the discriminator constitutes a dynamic "game process", and the images generated by the trained generator are finally "indistinguishable from the real", that is, infinitely close to the real image. Specifically in the embodiment of the present invention, the content image and the output image are input into the discriminator to train the discriminator, and the judgment value of the output image is obtained after each input, wherein the judgment value can be the probability that the output image belongs to the real image.

[0060] S107. Adjust network parameters of the generator and the discriminator according to the determination value, the output image, the content image, and the style image.

[0061] In an optional embodiment of the present invention, the total loss can be calculated based on the judgment value, the output image, the content image and the style image. Specifically, the structural loss of the content image and the output image, the content loss of the content image and the output image, and the style loss of the style image and the output image can be calculated. The style image is blurred to obtain a blurred style image. The style image and the blurred style image are input into the discriminator to obtain the judgment value of the style image and the judgment value of the blurred style image. The judgment value of the output image, the judgment value of the style image, and the sum of the squares of the judgment value of the blurred style image are calculated to obtain the adversarial loss. The edge loss of the style image and the output image is calculated. The structural loss, content loss, style loss, adversarial loss and edge loss are summed to obtain the total loss.

[0062] After each round of iterative training, the training of the generator and discriminator is stopped when the total loss is less than the preset value. Otherwise, the network parameters of the discriminator and generator are adjusted according to the total loss, and then a new round of training iteration update is started until the total loss is less than the preset value. After stopping the training, the adversarial generative network is obtained. When adjusting the network parameters, the gradient can be calculated according to the total loss to adjust the network parameters of the encoding network and decoding network of the generator, and the gradient can be calculated according to the adversarial loss to adjust the network parameters of the discriminator. After stopping the training, after inputting any content image into the generator of the adversarial generative network, the generator automatically injects the style momentum accumulation into the content image to obtain the image after the content image style transfer.

[0063] In practical applications, the network structure of the generator can be the same. A generator can be trained for each style transfer requirement, and the network parameters of the generator and the style momentum accumulation of the style can be saved. When the user needs to stylize the content image, the network parameters and style momentum accumulation of the generator corresponding to the style can be selected.

[0064] The generator in the embodiment of the present invention is provided with a style momentum accumulation amount. The style momentum accumulation amount is updated by extracting a style image of the same style and inputting it into a trained style feature extraction network. The generator only needs to input a content image to obtain an output image after style transfer, and then adjusts the network parameters of the generator and the discriminator according to the judgment value of the discriminator on the output image, the output image, the content image and the style image. After the training is completed, the final style momentum accumulation amount can contain the style information of the style. On the one hand, the style image is input into the trained style feature network to update the style momentum accumulation amount of the generator, and the content image is input into the generator to train the generator, thereby realizing the decoupling of content and style. The generator is more focused on injecting the style momentum accumulation amount into the content image to learn the style. It is easier to constrain the content and style in the generator, so that the image after style transfer can not only maintain the content of the content image, but also learn the style of the style image, thereby improving the stylization effect. On the other hand, there is no need to obtain a large number of stylized images of the content image as labels for supervised training. During training, it is sufficient to obtain the content image and multiple arbitrary style images of the same style, thereby reducing the difficulty of obtaining training data and making it easy to obtain training data.

[0065] Embodiment 2

[0066] Figure 2A A flowchart of a method for training a generative adversarial network is provided in the second embodiment of the present invention. The embodiment of the present invention is optimized on the basis of the first embodiment. Specifically, Figure 2A As shown, the generative adversarial network training method of the embodiment of the present invention may include the following steps:

[0067] S201: Acquire multiple content images and multiple style images of the same style.

[0068] In the embodiment of the present invention, the content image may be an image of the style of the image to be edited, and the style image may be an image of a certain style, wherein the style of the image may be a comic style, a martial arts style, an artistic style, etc., and each style may be further subdivided. When obtaining training data, any number of images may be obtained as content images, and then any number of images of the same style may be obtained as style images. In each round of training, a content image and a style image may be randomly extracted as training samples to train the generative adversarial network.

[0069] S202, initializing the network parameters of the discriminator of the generative adversarial network, initializing the style momentum accumulator, the encoding network and the decoding network parameters of the generator, and obtaining the trained residual network used in the generator.

[0070] In an embodiment of the present invention, the generative adversarial network includes a discriminator and a generator, and the generator may include an encoding network, a decoding network, and a residual network, wherein the residual network may be various pre-trained neural networks. The initialization referred to in the embodiment of the present invention may be to initialize the network parameters of the discriminator, initialize the network parameters of the encoding network and the decoding network of the generator, and initialize the style momentum accumulation of the generator, the style momentum accumulation includes the style information of the specified style, and the style momentum accumulation is used in the generator to be injected into the content image to obtain the image after the content image style transfer.

[0071] Specifically, initialization may be to construct a network structure of a discriminator, an encoding network, and a decoding network, and set network parameters of the network structure. In the embodiment of the present invention, the discriminator, the encoding network, and the decoding network may be various neural networks.

[0072] like Figure 2B FIG. 1 is a schematic diagram of each network in an embodiment of the present invention. Figure 2B In the figure, the generator 30 includes an encoding network 301, a decoding network 302, a residual network 303, a style accumulation module 304, and a conversion module 305, wherein the encoding network 301 and the decoding network 302 can be symmetrical convolutional neural networks and deconvolutional neural networks, the residual network 303 is connected between the encoding network 301 and the decoding network 302, the style feature extraction network 40 includes a downsampling convolution layer 401, a global pooling layer 402, and a fully connected layer 403, the style accumulation module 304 is used to update the style momentum accumulation according to the style feature 60 output by the style feature extraction network 40, the conversion module 305 is used to convert the style momentum accumulation into a mean and a variance, and the residual network 303 is used to inject the mean and the variance into the content feature 50.

[0073] S203: extract the style image and input it into a pre-trained style feature extraction network to obtain style features.

[0074] like Figure 2B As shown, in an example of an embodiment of the present invention, the pre-trained style feature extraction network 40 can be a convolutional neural network CNN, a recurrent neural network RNN, a deep neural network DNN, etc. The style feature extraction network 40 includes a downsampling convolution layer 401, a global pooling layer 402, and a fully connected layer 403. A style image 20 can be randomly extracted from multiple style images of the same style and input into the style feature extraction network 40. After the style image 20 passes through the downsampling convolution layer 401, the global pooling layer 402, and the fully connected layer 403, the style feature 60 is output.

[0075] S204: Update the style momentum accumulation according to the style feature.

[0076] In an optional embodiment of the present invention, the style feature may be converted into a style implicit variable, and then the style momentum accumulation amount may be updated according to a preset attenuation coefficient and the style implicit variable. Specifically, Figure 2B As shown, the style feature 60 may be a style implicit variable, which includes the style information of the style image. The style accumulation module 304 maintains a style momentum accumulation. When the style feature extraction network 40 outputs the style implicit variable, the style momentum accumulation may be updated by the following formula:

[0077] x acc =decay×x acc +(1-decay)×x

[0078] In the above formula, x acc is the style momentum accumulation, decay coefficient, the value can be 0.999, x is the style implicit variable, style momentum accumulation x acc It is a non-training parameter during training. After the training of the generative adversarial network is completed, it is saved as a parameter of the generator. When the generator is applied to transfer the style of the content image, the style accumulation module 304 directly returns the style momentum accumulation x acc , x acc It contains the style information of a certain style learned during training, and can be directly applied to the content feature map of the content image. When the generator is applied after training, there is no need to provide a style image to transfer the style of the content image, which avoids the extraction of style features from the style image, saves the calculation time of the generator, and improves the efficiency of image style transfer.

[0079] S205: Use the encoding network to encode the content image to obtain a content feature map of the content image.

[0080] In an embodiment of the present invention, the content image may be preprocessed to obtain a preprocessed content image, and then the preprocessed content image may be input into an encoding network to obtain a content feature map, wherein the preprocessing includes adjusting the image size.

[0081] For example, Figure 2B As shown, the encoding network 301 can be a network including multiple downsampling convolution layers. After the content image 10 is cropped into an image of a specified size, the cropped content image is input into the downsampling convolution layer of the encoding network 301. Each downsampling convolution layer performs sampling and encoding processing on the cropped content image to output a downsampling feature map, and the downsampling feature map is input into the next downsampling convolution layer. The downsampling feature map output by the last downsampling convolution layer of the encoding network 301 is the content feature map F. H×W×D , H and W are the height and width of the content feature map respectively, and D is the number of channels, such as Figure 2B As shown, the encoding network 301 finally outputs a content feature map 50.

[0082] S206: Input the content feature map and the style momentum accumulation into the residual network to obtain a style transfer feature map.

[0083] In an optional embodiment of the present invention, the style momentum accumulation can be converted to obtain the mean and variance of the style momentum accumulation, and the mean and variance of the style momentum accumulation and the content feature map are input into the residual network to migrate the style momentum accumulation to the content feature map through the residual network to obtain a style migration feature map.

[0084] like Figure 2B As shown, the style momentum accumulation output by the style accumulation module 304 can pass through a conversion module 305 (multi-layer perceptron) to output the mean μ and variance σ of the style momentum accumulation. The mean μ and variance σ of the style momentum accumulation and the content feature map 50 are input into the residual network 303 together to obtain the style transfer feature map.

[0085] In an optional embodiment of the present invention, the residual network may be an adaptive instance normalized residual network (AdaINResBlk), and the residual network may describe a style image as the mean and variance of a style feature map, and implement style injection by changing the mean and variance of a content feature map of a content image. With x representing the content feature map and y representing the style image, the formula of the residual network is:

[0086]

[0087] In the above formula, μ and σ are the mean and variance of the style feature map to be injected. Specifically in the embodiment of the present invention, x is the content feature, y is the style feature, and AdaIN(x, y) is the style migration feature map.

[0088] The embodiment of the present invention can inject the style features of the style image into the content feature map of the content image through the residual network, thereby replacing the style of the content image with the style of the style image, and can retain the content information such as the posture and structure of the target in the content image, thereby realizing the combination of the content features of the content image and the style features of the style image.

[0089] S207: Using the decoding network to decode the style transfer feature map to obtain an output image.

[0090] Specifically, Figure 2B As shown, the decoding network 302 can be an upsampling convolutional neural network. The residual network 303 outputs the style transfer feature map and then inputs it into the decoding network 302 for upsampling, and finally obtains an output image with the same size as the content image 10. The output image 70 retains the content of the content image 10 and has the style of the style image 20, as shown in FIG. Figure 2B As shown, the person in the output image 70 is the person in the content image 10 , and the identity, posture, expression and other contents of the person remain unchanged, and the style is the style in the style image 20 .

[0091] S208: Input the content image and the output image into the discriminator for training to obtain a judgment value of the output image.

[0092] The embodiment of the present invention alternately trains the generator and the discriminator. The generator is first trained to obtain an output image. The output image and the content image are then used to train the discriminator. Then, the generator is trained alternately. The generative adversarial network is trained alternately in this way. Training the generator once and the discriminator once is one round of training. After each round of training, the generator generates an output image. The discriminator judges the output image to obtain a judgment value. The judgment value can be the probability that the output image belongs to a real image.

[0093] S209 , calculating a total loss according to the determination value of the output image, the output image, the content image, and the style image.

[0094] In the embodiment of the present invention, the total loss may be the sum of the structure loss, the content loss, the style loss, the adversarial loss and the edge loss, and the total loss may be calculated by the following sub-steps:

[0095] S2091. Calculate the structural losses of the content image and the output image.

[0096] Optionally, the content image and the output image may be sequentially input into a pre-trained image structure feature extraction network to obtain the structure features of the content image and the structure features of the output image, and the distance between the structure features of the content image and the structure features of the output image is calculated to obtain the structure loss, wherein the structure features of the image may be features of key points of the target in the image, for example, if the target is a person, the structure features may be key points of the face of the person, key points of the hand, etc., which may express features such as the identity and posture of the person, and the output image out_img and the content image source_img may be respectively input into a pre-trained image structure feature extraction network VGG to obtain the structure feature FeatVGG of the output image out_img. out And the structural features of the content image source_img FeatVGG src , the structural loss can be calculated by the following formula:

[0097] percp_loss=||FeatVGG out -FeatVGG src || 2

[0098] The embodiment of the present invention constrains the training of the generator by calculating the structural loss, so that the structure of the output image after style transfer output by the generator is similar to that of the content image, so that the output image can better maintain the structure, posture and other contents of the target in the content image.

[0099] S2092: Calculate content loss of the content image and the output image, and calculate style loss of the style image and the output image.

[0100] Optionally, the content image and the output image may be sequentially input into the encoding network of the generator to obtain the content features of the content image and the content features of the output image, and the distance between the content features of the content image and the content features of the output image is calculated to obtain the content loss. The specific calculation formula is as follows:

[0101]

[0102] In the above formula, content_loss is the content loss, Feat content is the content feature of the content image, is the content feature of the output image.

[0103] For style loss, the style image and the output image are input into the style feature extraction network in sequence to obtain the style features of the style image and the style features of the output image. The distance between the style features of the style image and the style features of the output image is calculated to obtain the style loss. The specific calculation formula is as follows:

[0104]

[0105] In the above formula, style_loss is the style loss, Feat style is the style feature of the style image, is the style feature of the output image.

[0106] The embodiment of the present invention constrains the generator by calculating the content loss and the style loss, so that the output image can not only perfectly maintain the content of the content image, but also better maintain the style of the style image.

[0107] S2093: Perform blur processing on the style image to obtain a blurred style image.

[0108] In an embodiment of the present invention, the style image may be subjected to Gaussian blur processing to obtain a style image after Gaussian blur processing. By performing Gaussian blur processing on the style image, the generator may better learn line information in the style image.

[0109] S2094: Input the style image and the blurred style image into the discriminator to obtain a determination value of the style image and a determination value of the blurred style image.

[0110] Let G be the generator, D be the discriminator, and the judgment value of the style image is (D(IMG style )-1) 2 , the judgment value of the style image after blur processing is D(GaussianBlur(IMG style )) 2 , among which, IMG style For the style image, GaussianBlur (IMG style ) is the style image after Gaussian blur processing.

[0111] S2095. Calculate the sum of the squares of the judgment value of the output image, the judgment value of the style image, and the judgment value of the blurred style image to obtain an adversarial loss.

[0112] The specific formula is as follows:

[0113] gan_loss=(D(IMG style )-1) 2 +D(G(IMG style ,IMG content )) 2 +D(GaussianBlur(IMG style )) 2

[0114] In the above formula, gan_loss is the adversarial loss, D(G(IMG style ,IMG content )) 2 The judgment value of the output image is obtained by constraining the training of the generator and the discriminator through adversarial loss, so that the generator can better learn the image style and the generated output image can obtain better style effect.

[0115] S2096: Calculate the edge loss of the style image and the output image.

[0116] Specifically, the style image and the output image can be sequentially input into the edge extraction model to obtain the edge features of the style image and the edge features of the output image, and the distance between the edge features of the style image and the edge features of the output image is calculated to obtain the edge loss. The specific calculation formula is as follows:

[0117] edge_loss=||EdgeNet(IMG style )–EdgeNet(G(IMG style ,IMG content ))|| 2

[0118] In the above formula, EdgeNet(IMG style ) is the edge feature of the style image, EdgeNet(G(IMG style ,IMG content )) is the edge feature of the output image. The embodiment of the present invention constrains the generator through edge loss, so that the main edge of the image output by the generator after style transfer is stronger and clearer, and at the same time can eliminate the influence of non-main lines, thereby improving the style transfer effect.

[0119] S2097. Calculate and sum the structure loss, the content loss, the style loss, the adversarial loss, and the edge loss to obtain a total loss.

[0120] Specifically, the total loss can be the sum of structure loss, content loss, style loss, adversarial loss and edge loss, as shown in the following formula:

[0121] total_loss=perp_loss+contet_loss+style_loss+gan_loss+edge loss

[0122] Of course, in practical applications, weights can also be set for structural loss, content loss, style loss, adversarial loss and edge loss, and the product of the weight and the loss can be calculated to obtain the weight of each loss, and the sum of the weights can be used as the total loss. Those skilled in the art can also calculate the weighted average of each loss as the total loss. The embodiment of the present invention does not limit the method of calculating the total loss.

[0123] S210: Determine whether the total loss is less than a preset threshold.

[0124] Optionally, after a round of alternating training, the total loss is calculated to determine whether the total loss is less than a preset threshold. If the total loss is less than the preset threshold, it means that the accuracy of the generator is high enough and the output image generated by the generator is sufficient to deceive the discriminator. The training of the generator and the discriminator can be stopped and S211 can be executed. If the total loss is greater than the preset threshold, it means that the accuracy of the generator is insufficient and the discriminator can still identify the authenticity of the output image generated by the generator. Then S212 can be executed to adjust the parameters of the discriminator according to the adversarial loss, and adjust the parameters of the encoding network and the decoding network in the generator according to the total loss.

[0125] S211. Stop training the generator and the discriminator.

[0126] That is, stop training the generator and discriminator, save the network parameters of the encoding network and decoding network of the generator, and save the style momentum accumulation output by the style accumulation module in the generator.

[0127] S212. Adjust the parameters of the discriminator according to the adversarial loss, and adjust the parameters of the encoding network and the decoding network in the generator according to the total loss.

[0128] Specifically, the parameters of the discriminator, the encoding network, and the decoding network are adjusted by updating the parameters through a gradient descent algorithm, wherein the gradient descent algorithm may be a stochastic gradient descent method SGD or other gradient descent methods. The embodiments of the present invention do not limit the gradient algorithm and the method for adjusting and updating the parameters.

[0129] The embodiment of the present invention obtains multiple content images and multiple style images of the same style, initializes the network parameters of the discriminator of the generative adversarial network, initializes the style momentum accumulation of the generator, and the network parameters of the encoding network and the decoding network, and after obtaining the trained residual network used in the generator, extracts the style image and inputs it into a pre-trained style feature extraction network to obtain the style feature, updates the style momentum accumulation according to the style feature, uses the encoding network to encode the content image to obtain the content feature map of the content image, inputs the content feature map and the style momentum accumulation into the residual network to obtain the style transfer feature map, and uses the decoding network to decode the style transfer feature map to obtain the output image. On the one hand, the style image is input into the trained style feature network to update the style momentum accumulation of the generator, and the content image is input into the generator to train the generator, thereby realizing the decoupling of content and style. The generator is more focused on injecting the style momentum accumulation into the content features in the content graph to learn the style. It is easier to constrain the content and style in the generator, so that the image after style transfer can not only maintain the content of the content image, but also learn the style of the style image. On the other hand, there is no need to obtain a large number of training images stylized with content images as labels for supervised training. It is only necessary to obtain content images and multiple arbitrary style images of the same style, and it is easy to obtain training data.

[0130] The total loss includes structural loss, content loss, style loss, adversarial loss and edge loss. The total loss is used to adjust the network parameters of the generator so that the output image after style transfer has a realistic style effect and can maintain the content of the content image.

[0131] Embodiment 3

[0132] Figure 3 This is a flowchart of a method for image style transfer provided in Embodiment 3 of the present invention. The embodiment of the present invention is applicable to the case of changing the image style. The method can be performed by an image style transfer device in the embodiment of the present invention. The image style transfer device can be implemented by hardware or software and integrated in the electronic device provided in the embodiment of the present invention. Specifically, Figure 3 As shown, the image style transfer method of the embodiment of the present invention may include the following steps:

[0133] S301: Obtain an image to be processed.

[0134] In an embodiment of the present invention, the image to be processed may be an image whose image style is to be changed, and the image may be any image. In an application scenario, the content image may be a selfie image of a user. Of course, the content image may also be an image in other application scenarios, for example, it may be a user's avatar in an instant messaging application, an image in a short video, and so on.

[0135] S302: Determine a generator in response to the image style selected by the user, wherein the generator is used to transfer the image style selected by the user to the image to be processed.

[0136] In an embodiment of the present invention, an interactive interface may be provided, in which the user is provided with the function of determining content images and selecting image styles. The user may specify content images and select image styles in the interactive interface. By way of example, the interactive interface may provide a content image upload operation, and the user may upload content images in the interactive interface. For example, the interactive interface may prompt the user to upload content images. Of course, the content images specified by the user may also be obtained through other interactive operations, and this is not limited by the embodiment of the present invention.

[0137] At the same time, a variety of image styles are provided for users to choose from in the interactive interface. When the user determines the image style, the network parameters of the generator and the style momentum accumulation corresponding to the image style selected by the user can be obtained. Among them, the network parameters of the generator and the style momentum accumulation corresponding to the image style selected by the user can be obtained through the generative adversarial network training method of Example 1 or Example 2 of the present invention. For details, please refer to Example 1 or Example 2, which will not be described in detail here.

[0138] S303: Input the image to be processed into the generator to obtain an image after style transfer.

[0139] Specifically, the image to be processed is input into the generator, in which the encoding network extracts content features from the image to be processed, the residual network injects the mean and variance of the style momentum accumulation corresponding to the image style selected by the user into the content features to obtain the style transfer features, and the decoding network decodes the style transfer features to obtain a style-transferred image with the same size as the image to be processed.

[0140] The image style transfer method of the embodiment of the present invention can determine a corresponding generator in response to the image style selected by the user. The generator can transfer the style selected by the user to the content image without the user selecting the style image, thus avoiding the extraction of style features from the style image, saving the calculation time of the generator, and improving the efficiency of image style transfer.

[0141] The generator of the embodiment of the present invention is provided with a style momentum accumulation. During each iteration of the generative adversarial network during the training process, the style momentum accumulation is updated by extracting a style image of the same style and inputting it into the trained style feature extraction network. The generator only needs to input the content image to obtain the output image after style transfer, and then adjust the network parameters of the generator and the discriminator according to the judgment value of the discriminator on the output image, the output image, the content image and the style image, so that the final style momentum accumulation can contain the style features of the style. On the one hand, the style image is input into the trained style feature network to update the style momentum accumulation of the generator, and the content image is input into the generator to train the generator, thereby realizing the decoupling of content and style. The generator is more focused on injecting the style momentum accumulation into the content features in the content image to learn the style. It is easier to constrain the content and style in the generator, so that the image after style transfer can not only maintain the content of the content image, but also learn the style of the style image. On the other hand, there is no need to obtain a large number of images stylized by the training content image as labels for supervised training. It is sufficient to obtain the content image and multiple arbitrary style images of the same style, and the training data is easy to obtain.

[0142] Embodiment 4

[0143] Figure 4 is a structural block diagram of a generative adversarial network training device provided by Embodiment 4 of the present invention, such as Figure 4 As shown, the generative adversarial network training device of the embodiment of the present invention may specifically include the following modules:

[0144] A training data acquisition module 801 is used to acquire multiple content images and multiple style images of the same style;

[0145] An initialization module 802 is used to initialize the network parameters of the generator and the discriminator of the generative adversarial network, and to initialize the style momentum accumulation of the generator;

[0146] A style feature extraction module 803 is used to extract the style features of the style image and input it into a pre-trained style feature extraction network to obtain the style features;

[0147] A style momentum accumulation updating module 804, configured to update the style momentum accumulation according to the style feature;

[0148] A generator training module 805 is used to extract the content image and input it into the generator, so as to inject the style momentum accumulation into the content image through the generator to obtain an output image after style transfer;

[0149] A discriminator training module 806, configured to input the content image and the output image into the discriminator for training to obtain a judgment value;

[0150] The network parameter adjustment module 807 is used to adjust the network parameters of the generator and the discriminator according to the determination value, the output image, the content image and the style image.

[0151] The generative adversarial network training device provided in the embodiment of the present invention can execute the generative adversarial network training method provided in the first and second embodiments of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0152] Embodiment 5

[0153] Figure 5 is a structural block diagram of an image style transfer device provided by Embodiment 5 of the present invention, such as Figure 5 As shown, the image style transfer device of the embodiment of the present invention may specifically include the following modules:

[0154] The image to be processed acquisition module 501 is used to acquire the image to be processed;

[0155] A generator determination module 502 is used to determine a generator in response to an image style selected by a user, wherein the generator is used to transfer the image style selected by the user to the image to be processed;

[0156] A style transfer module 503, used for inputting the image to be processed into the generator to obtain an image after style transfer;

[0157] Wherein, the generator is trained by the generative adversarial network training method described in any one of claims 1-10.

[0158] The image style transfer device provided in the embodiment of the present invention can execute the image style transfer method provided in the third embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0159] Embodiment 6

[0160] Reference Figure 6 , shows a schematic diagram of the structure of an electronic device in an example of the present invention. Figure 6 As shown, the electronic device may specifically include: a processor 601, a storage device 602, a display screen 603 with a touch function, an input device 604, an output device 605, and a communication device 606. The number of processors 601 in the device may be one or more. Figure 6 A processor 601 is taken as an example. The processor 601, storage device 602, display screen 603, input device 604, output device 605 and communication device 606 of the device can be connected through a bus or other means. Figure 6The device is used to execute the generative adversarial network training method and / or the image style transfer method provided in any embodiment of the present invention.

[0161] An embodiment of the present invention also provides a computer-readable storage medium. When the instructions in the storage medium are executed by a processor of an electronic device, the electronic device can execute the generative adversarial network training method and / or the image style transfer method as described in the above method embodiments.

[0162] It should be noted that, for the device, electronic device, and storage medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0163] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0164] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A generative adversarial network training method, characterized in that: include: Get multiple content images and multiple style images of the same style; Initialize the network parameters of the generator and discriminator of the generative adversarial network, and initialize the style momentum accumulation of the generator; Extracting the style image and inputting it into a pre-trained style feature extraction network to obtain style features; updating the style momentum accumulation amount according to the style feature; Extracting the content image and inputting it into the generator, so as to inject the style momentum accumulation into the content image through the generator to obtain an output image after style transfer; Inputting the content image and the output image into the discriminator for training to obtain a judgment value of the output image; The network parameters of the generator and the discriminator are adjusted according to the decision value, the output image, the content image and the style image.

2. The method according to claim 1, characterized in that Initializing the network parameters of the generator and the discriminator of the generative adversarial network, and initializing the style momentum accumulation of the generator, include: Initialize the network parameters of the discriminator of the generated adversarial network; Initialize the style momentum accumulator of the generator, the network parameters of the encoding network and the decoding network, and obtain the trained residual network used in the generator.

3. The method according to claim 1, characterized in that The updating of the style momentum accumulation according to the style feature comprises: Converting the style features into style latent variables; The style momentum accumulation amount is updated according to a preset attenuation coefficient and the style implicit variable.

4. The method according to claim 2, characterized in that: The extracting the content image and inputting it into the generator, so as to inject the style momentum accumulation into the content image through the generator to obtain an output image after style transfer, comprises: Using the encoding network to encode the content image to obtain a content feature map of the content image; Inputting the content feature map and the style momentum accumulation into the residual network to obtain a style transfer feature map; The decoding network is used to decode the style transfer feature map to obtain an output image.

5. The method according to claim 4, characterized in that The step of inputting the content feature map and the style momentum accumulation into the residual network to obtain a style transfer feature map includes: Converting the style momentum accumulation to obtain a mean and a variance of the style momentum accumulation; The mean and variance of the style momentum accumulation and the content feature map are input into the residual network, so as to transfer the style momentum accumulation to the content feature map through the residual network to obtain a style transfer feature map.

6. The method according to any one of claims 2 to 5, characterized in that: The adjusting the parameters of the generator and the discriminator according to the determination value, the output image, the content image and the style image comprises: calculating a total loss based on the determination value of the output image, the output image, the content image, and the style image; Determining whether the total loss is less than a preset threshold; If yes, stop training the generator and the discriminator; If not, the parameters of the generator and the discriminator are adjusted according to the total loss, and the process returns to the step of extracting the style image and inputting it into a pre-trained style feature extraction network to obtain style features.

7. The method according to claim 6, characterized in that The calculating the total loss according to the determination value, the output image, the content image and the style image comprises: Calculating a structural loss of the content image and the output image; Calculating a content loss of the content image and the output image, and calculating a style loss of the style image and the output image; Performing blur processing on the style image to obtain a blurred style image; Inputting the style image and the blurred style image into the discriminator to obtain a determination value of the style image and a determination value of the blurred style image; Calculating the sum of squares of the judgment value of the output image, the judgment value of the style image, and the judgment value of the blurred style image to obtain an adversarial loss; Calculating the edge loss of the style image and the output image; The total loss is obtained by calculating and summing the structure loss, the content loss, the style loss, the adversarial loss, and the edge loss.

8. The method according to claim 7, characterized in that The calculating the structural loss of the content image and the output image comprises: Inputting the content image and the output image into a pre-trained image structure feature extraction network in sequence to obtain the structure features of the content image and the structure features of the output image; The distance between the structural feature of the content image and the structural feature of the output image is calculated to obtain a structural loss.

9. The method according to claim 7, characterized in that: The calculating the content loss of the content image and the output image, and the calculating the style loss of the style image and the output image, include: Inputting the content image and the output image into the encoding network of the generator in sequence to obtain content features of the content image and content features of the output image; Inputting the style image and the output image into the style feature extraction network in sequence to obtain style features of the style image and style features of the output image; Calculating the distance between the content feature of the content image and the content feature of the output image to obtain content loss; The distance between the style feature of the style image and the style feature of the output image is calculated to obtain a style loss.

10. The method according to claim 7, characterized in that The calculating the edge loss of the style image and the output image includes: Inputting the style image and the output image into an edge extraction model in sequence to obtain edge features of the style image and edge features of the output image; The distance between the edge feature of the style image and the edge feature of the output image is calculated to obtain the edge loss.

11. A method for image style transfer, characterized in that: include: Get the image to be processed; Determining a generator in response to the image style selected by the user, the generator being used to transfer the image style selected by the user to the image to be processed; Inputting the image to be processed into the generator to obtain an image after style transfer; Wherein, the generator is trained by the generative adversarial network training method described in any one of claims 1-10.

12. A generative adversarial network training device, characterized in that: include: A training data acquisition module, used to acquire multiple content images and multiple style images of the same style; An initialization module, used to initialize the network parameters of the generator and the discriminator of the generative adversarial network, and to initialize the style momentum accumulation of the generator; A style feature extraction module, used to extract the style features of the style image and input it into a pre-trained style feature extraction network; A style momentum accumulation updating module, used for updating the style momentum accumulation according to the style feature; A generator training module, used for extracting the content image and inputting it into the generator, so as to inject the style momentum accumulation into the content image through the generator to obtain an output image after style transfer; A discriminator training module, used for inputting the content image and the output image into the discriminator for training to obtain a judgment value; A network parameter adjustment module is used to adjust the network parameters of the generator and the discriminator according to the determination value, the output image, the content image and the style image.

13. An image style transfer device, characterized in that: include: The module for acquiring the image to be processed is used to acquire the image to be processed; A generator determination module, configured to determine a generator in response to an image style selected by a user, wherein the generator is configured to transfer the image style selected by the user to the image to be processed; A style transfer module, used for inputting the image to be processed into the generator to obtain an image after style transfer; Wherein, the generator is trained by the generative adversarial network training method described in any one of claims 1-10.

14. An electronic device, characterized in that: The electronic device comprises: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the generative adversarial network training method as described in any one of claims 1-10, and / or the image style transfer method as described in claim 11.

15. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, it implements the generative adversarial network training method as described in any one of claims 1 to 10, and / or the image style transfer method as described in claim 11.

Images