Image stylized model training method and device and computer equipment

Abstract

The invention provides an image stylized model training method and device and computer equipment, and belongs to the technical field of computers. The method comprises the steps of training a first initial stylization model based on a plurality of first sample images to obtain a first target stylization model, and performing random stylization processing on a plurality of second sample images through the first target stylization model to obtain first stylization data; training a second initial stylized model based on a preset style label, each second sample image and a stylized image in the first stylized data to obtain a second target stylized model; performing stylization processing on the plurality of third sample images through a second target stylization model to obtain each piece of paired data; and training the third initial stylized model based on the pairing data to obtain a third target stylized model. The efficiency of training the model and the performance of the model obtained by training can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, apparatus and computer device for training a stylized model of an image. Background Art

[0002] With the development of computer technology, relevant technicians have developed algorithms that can convert image styles. Specifically, they can convert real character images into images of different styles such as anime style and hand-painted style.

[0003] In the related art, it is generally necessary to establish a stylized model based on a generative adversarial network (GAN) to achieve stylized conversion of images. GAN includes a generative model and a discriminative model. Specifically, the preset Pix2PixGAN can be trained with thousands of training data, and the training data is generally paired data, to obtain a high-quality generative model. Then the stylized image generated by the generative model is input into the discriminative model to achieve alternating optimization training of the generative model and the discriminative model, and finally a model that can output high-quality stylized images can be obtained.

[0004] However, since the solution of the related art requires a large amount of paired data to train the generated model, it takes a lot of time and manpower to collect the training data. If the amount of the training data is small or the quality is poor, the performance of the trained model will be poor. Therefore, this solution has the problem of not being able to balance the efficiency of the training model and the performance of the model. Summary of the invention

[0005] The purpose of this application is to provide a method, device and computer equipment for training a stylized model of an image, which can improve the efficiency of the training model and the performance of the trained model.

[0006] The embodiment of the present application is implemented as follows:

[0007] In a first aspect of an embodiment of the present application, a method for training a stylized model of an image is provided, the method comprising:

[0008] A first initial stylization model is trained based on a plurality of first sample images to obtain a first target stylization model, and a plurality of second sample images are randomly stylized by the first target stylization model to obtain a plurality of first stylized data; each of the first sample images includes a stylized face, each of the second sample images includes a real face image, each of the first stylized data includes a stylized image and a style label, and the number of each of the first stylized data is greater than the number of each of the first sample images;

[0009] Training a second initial stylized model based on a preset style label, the second sample image, and the stylized image in the first stylized data to obtain a second target stylized model, wherein the preset style label is a style label in each of the first stylized data;

[0010] Performing stylization processing on a plurality of third sample images by using the second target stylization model to obtain a plurality of paired data, each of the paired data comprising: a real face image and a corresponding stylized image of a preset style;

[0011] The third initial stylized model is trained based on each of the paired data to obtain a third target stylized model.

[0012] Optionally, the training of the second initial stylized model based on the preset style label, the second sample image, and the stylized image in the first stylized data to obtain the second target stylized model includes:

[0013] The second initial stylized model outputs a first generation result based on the preset style label and the second sample image, and determines first loss information according to the second sample image, the first generation result and the stylized face in the first stylized data;

[0014] The second initial stylized model is iteratively optimized according to the first loss information until a first preset condition is satisfied, and the second initial stylized model that satisfies the first preset condition is used as the second target stylized model.

[0015] Optionally, determining first loss information according to the second sample image, the first generation result, and a stylized face in the first stylized data includes:

[0016] Extracting feature information of the first generated result and feature information of a real face image in the second sample image based on a VGG model, and determining a content loss value based on the feature information of the first generated result and the feature information of the real face image in the second sample image;

[0017] Respectively converting the first generated result and the stylized face in the first stylized data into a first grayscale image and a second grayscale image, and extracting feature information of the first grayscale image and feature information of the second grayscale image based on the VGG model;

[0018] Determine a texture loss value based on the feature information of the first grayscale image and the feature information of the second grayscale image;

[0019] Determine a first generative adversarial loss value based on the real face images in each of the second sample images, the stylized faces in each of the first stylized data, and the generator and the discriminator in the second initial stylized model;

[0020] The first loss information is determined according to the content loss value, the texture loss value and the first generative adversarial loss value.

[0021] Optionally, performing stylization processing on a plurality of third sample images by using the second target stylization model to obtain a plurality of paired data includes:

[0022] Inputting each of the third sample images into the second target stylization model to obtain target stylization data output by the second target stylization model;

[0023] Based on the mapping relationship between each of the third sample images and the corresponding target stylized data, each of the pairing data is obtained.

[0024] Optionally, the training a third initial stylized model based on each of the paired data to obtain a third target stylized model includes:

[0025] Outputting a second generation result by the third initial stylization model according to the pairing data, and determining second loss information according to the second generation result;

[0026] The third initial stylized model is iteratively optimized according to the second loss information until a second preset condition is satisfied, and the third initial stylized model that satisfies the second preset condition is used as the third target stylized model.

[0027] Optionally, determining the second loss information according to the second generation result includes:

[0028] Determine a reconstruction loss value according to the second generated result and the target stylized data in the paired data;

[0029] Determine a first perceptual loss value according to feature information of each of the target stylized data, feature information of each of the second generated results, and a preset feature parameter;

[0030] Determine a second generative adversarial loss value based on each of the third sample images, each of the second generated results, each of the target stylized data, and a generator and a discriminator in the third initial stylized model;

[0031] Second loss information is determined according to the reconstruction loss value, the first perceptual loss value, and the second generative adversarial loss value.

[0032] Optionally, the training of the first initial stylization model based on the plurality of first sample images to obtain the first target stylization model includes:

[0033] Performing a preprocessing operation on each of the first sample images, and using the preprocessed first sample images as each of the first sample images, wherein the preprocessing operation includes at least one of the following: face alignment processing and normalization processing;

[0034] Outputting third generation results by the first initial stylization model based on the first sample images;

[0035] Determine a third generative adversarial loss value based on the random latent vector, each of the first sample images, the generator and the discriminator in the first initial stylization model;

[0036] Determine a second perceptual loss value according to the feature information of each of the first sample images, the feature information of each of the third generated results, and a preset feature parameter;

[0037] Determine third loss information according to the third generative adversarial loss value and the second perceptual loss value;

[0038] The first initial stylized model is iteratively optimized according to the third loss information until a third preset condition is satisfied, and the first initial stylized model that satisfies the third preset condition is used as the first target stylized model.

[0039] According to a second aspect of an embodiment of the present application, a stylized model training device for an image is provided, the device comprising:

[0040] A first execution module is configured to train a first initial stylization model based on a plurality of first sample images to obtain a first target stylization model, and to perform random stylization processing on a plurality of second sample images through the first target stylization model to obtain a plurality of first stylized data; each of the first sample images includes a stylized face, each of the second sample images includes a real face image, each of the first stylized data includes a stylized image and a style label, and the number of each of the first stylized data is greater than the number of each of the first sample images;

[0041] a second execution module, configured to train a second initial stylized model based on a preset style label, each of the second sample images, and a stylized image in the first stylized data to obtain a second target stylized model, wherein the preset style label is a style label in each of the first stylized data;

[0042] The second execution module is further used to perform stylization processing on a plurality of third sample images through the second target stylization model to obtain a plurality of paired data, each of the paired data comprising: a real face image and a corresponding stylized image of a preset style;

[0043] The third execution module is used to train the third initial stylization model based on each of the pairing data to obtain a third target stylization model.

[0044] According to a third aspect of an embodiment of the present application, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed by the processor, the image stylization model training method described in the first aspect is implemented.

[0045] According to a fourth aspect of an embodiment of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the image stylization model training method described in the first aspect is implemented.

[0046] The beneficial effects of the embodiments of the present application include:

[0047] The embodiment of the present application provides a method for training a stylized model of an image, wherein a first target stylized model is obtained by training a first initial stylized model based on a plurality of first sample images, and a plurality of second sample images are subjected to random stylization processing by the first target stylized model to obtain a plurality of first stylized data. In this way, the first target stylized model can be obtained based on a small number of first sample images, and then a large number of second sample images can be subjected to random stylization processing by the first target stylized model to obtain a large number of first stylized data.

[0048] The second initial stylized model is trained based on the preset style label, each second sample image, and the stylized image in the first stylized data to obtain a second target stylized model. Since the preset style label can indicate the style type learned by the second initial stylized model, the performance of the second target stylized model in stylizing an image of a specific style type can be improved. In addition, there is no need to use a large amount of paired data to train the second target stylized model, which can improve the efficiency of training the second target stylized model.

[0049] The second target stylized model is used to perform stylized processing on the plurality of third sample images to obtain a plurality of paired data. Since the number of each paired data is equal to or greater than the number of each third sample image, and the quality of the stylized image output by the second target stylized model is relatively high, a large amount of high-quality paired data can be obtained for subsequent processing.

[0050] The third initial stylized model is trained based on each paired data to obtain a third target stylized model. Since each paired data is a large amount of data generated by stylizing each third sample image with the trained second target stylized model, it is not necessary to spend a lot of time and manpower to collect a large amount of paired data. In addition, the quality of the stylized image of the preset style in each paired data is high, and the real face image in each paired data is a high-definition face image in FFHQ and / or CelebA, so the quality of each paired data can be ensured to be high.

[0051] That is, the image stylization model training method provided in the embodiment of the present application can improve the efficiency of training the model while ensuring the performance of the trained model.

[0052] In this way, the efficiency of training the model and the performance of the trained model can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0054] Figure 1 A flowchart of stylized model training for the first type of image provided in an embodiment of the present application;

[0055] Figure 2 A flowchart of stylized model training for the second image provided in an embodiment of the present application;

[0056] Figure 3 A flowchart of stylized model training for the third type of image provided in an embodiment of the present application;

[0057] Figure 4 A flowchart of stylized model training for the fourth type of image provided in an embodiment of the present application;

[0058] Figure 5 A flowchart of stylized model training for the fifth type of image provided in an embodiment of the present application;

[0059] Figure 6 A flowchart of stylized model training for the sixth type of image provided in an embodiment of the present application;

[0060] Figure 7 A flowchart of stylized model training for the seventh type of image provided in an embodiment of the present application;

[0061] Figure 8 A schematic diagram of the structure of a stylized model training device for an image provided in an embodiment of the present application;

[0062] Fig. 9 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0063] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations.

[0064] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for which protection is sought, but merely represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0065] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0066] In the description of the present application, it should be noted that the terms "first", "second", "third", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0067] In the related art, it is generally necessary to establish a stylized model based on GAN to achieve stylized conversion of images. GAN includes a generative model and a discriminative model. Specifically, the preset Pix2PixGAN can be trained with thousands of training data, and the training data is generally paired data, to obtain a high-quality generative model. Then the stylized image generated by the generative model is input into the discriminative model to achieve alternating optimization training of the generative model and the discriminative model, and finally a model that can output high-quality stylized images can be obtained.

[0068] However, since the solution of the related art requires a large amount of paired data to train the generated model, it takes a lot of time and manpower to collect the training data. If the amount of the training data is small or the quality is poor, the performance of the trained model will be poor. Therefore, this solution has the problem of not being able to balance the efficiency of the training model and the performance of the model.

[0069] To this end, an embodiment of the present application provides a method for training a stylized model of an image, wherein a first target stylized model is obtained by training a first initial stylized model based on a plurality of first sample images, and a plurality of second sample images are randomly stylized by the first target stylized model to obtain a plurality of first stylized data. A second initial stylized model is trained based on a preset style label, each of the second sample images, and a stylized image in the first stylized data to obtain a second target stylized model. A plurality of third sample images are stylized by the second target stylized model to obtain a plurality of paired data. A third initial stylized model is trained based on each paired data to obtain a third target stylized model. The efficiency of the training model and the performance of the trained model can be improved.

[0070] The present application embodiment is described by taking the stylized model training method of an image applied in a computer device as an example, but it does not mean that the present application embodiment can only be applied to the stylized model of the training image in a computer device.

[0071] The following is a detailed explanation of the image stylization model training method provided in the embodiment of the present application.

[0072] Figure 1 This is a flowchart of a method for training a stylized model of an image provided by the present application. The method can be applied to the above-mentioned computer device, which can be the aforementioned terminal device or server. Figure 1 , the embodiment of the present application provides a stylized model training method for an image, comprising:

[0073] Step 1001: training a first initial stylization model based on a plurality of first sample images to obtain a first target stylization model, and performing random stylization processing on a plurality of second sample images through the first target stylization model to obtain a plurality of first stylization data.

[0074] Optionally, each first sample image includes a stylized face.

[0075] For example, each first sample image may include various stylized character data collected by relevant technicians according to actual stylization needs. The number of each first sample image may be relatively small, generally 50-200 images.

[0076] Stylization may refer to converting an image into an image or data in any other style, such as anime style, oil painting style, hand-painted style, etc.

[0077] Optionally, the first initial stylization model may refer to an untrained neural network model for stylizing an image. The first initial stylization model may be a generative adversarial network based on the network architecture of StyleGAN2, wherein the first initial stylization model also includes a generator and a discriminator.

[0078] The first target stylized model may refer to a first initial stylized model that has been trained, that is, the performance of the first target stylized model is better than the first initial stylized model that has not been trained.

[0079] Optionally, each second sample image includes a real face image.

[0080] The second sample images may be face images in a corresponding high-definition face dataset (Flickr-Faces-Hight-Quality, FFHQ for short) or a celebrity face attribute dataset (CelebFaces Attribute, CelebA for short). Generally, the number of the second sample images may be much greater than the number of the first sample images.

[0081] Optionally, each first stylized data includes a stylized image and a style label.

[0082] The stylized image may be an image obtained after the first target stylization model performs random stylization processing on the second sample image.

[0083] The style tag may be used to indicate the style type of the stylized image. The style type may include any other style such as the above-mentioned cartoon style, oil painting style, hand-painted style, etc., which is not limited in the embodiment of the present application.

[0084] In addition, the random stylization process may refer to stylizing a real face without limiting the style type. That is, after performing random stylization on any second sample image, at least one stylized image of any possible style type corresponding to the second sample image may be obtained.

[0085] Optionally, the number of each first stylized data is greater than the number of each first sample image.

[0086] In addition, each first stylized data may be post-processed, for example, dark spot removal processing, AI tooth restoration processing, super-resolution processing, etc. may be performed on each first stylized data, which is not limited in the embodiment of the present application.

[0087] It is worth noting that if it is necessary to make the first target stylized model output a specific type of stylized data, then the stylized faces in each first sample image can be set to the specific type. For example, if it is necessary to make the first target stylized model output the above-mentioned anime-style stylized data, then the first sample images including anime-style faces can be used to train the first initial stylized model.

[0088] It is worth noting that by training the first initial stylization model through each first sample image, it can be ensured that the first target stylization model obtained after training can perform high-quality random stylization processing and output high-quality stylized data. In addition, the number of each second sample image is much larger than the number of each first sample image, so after performing random stylization processing on each second sample image through the first target stylization model, the first stylized data of the number far exceeding that of each first sample image can be obtained.

[0089] In this way, there is no need to spend a lot of time and manpower to collect a large number of first sample images. Instead, a first target stylized model can be obtained by training a small number of first sample images, and then a large number of real face images can be randomly stylized by the first target stylized model to obtain a large amount of stylized data or stylized images.

[0090] Step 1002: training a second initial stylized model based on a preset style label, each second sample image, and the stylized image in the first stylized data to obtain a second target stylized model.

[0091] Optionally, the preset style label is a style label in each first stylized data. The preset style label can be used to indicate the style type that the second initial stylized model needs to learn.

[0092] Optionally, the second initial stylization model may refer to a neural network model that has not been trained and is used to perform stylization processing of a specific style type on an image.

[0093] Specifically, the second initial stylization model may also be a generative adversarial network, wherein the second initial stylization model also includes a generator and a discriminator. The generator in the second initial stylization model may be an encoder-decoder network, and the discriminator in the second initial stylization model may be a PatchGAN architecture.

[0094] The second target stylized model may refer to a trained second initial stylized model, that is, the performance of the second target stylized model is better than that of a second initial stylized model that has not been trained.

[0095] It is worth noting that, since the preset style label can indicate the style type learned by the second initial stylization model, if the second initial stylization model needs to learn a certain style type during the training process, the stylized image in the first stylization data whose style label matches the certain style type can be input to the second initial stylization model to ensure that the obtained second target stylization model can subsequently output the certain style type. This can improve the performance of the second target stylization model in stylizing images of a specific style type.

[0096] In addition, since the preset style labels input when training the second initial stylized model, the stylized images in the first stylized data and the second sample images do not need to correspond one to one, that is, during the training process, the second initial stylized model can mainly learn the content information of each second sample image and the stylized features of the stylized images in the first stylized data, so that the second initial stylized model can be trained in an unpaired data manner.

[0097] In this way, there is no need to use a large amount of paired data to train the second target stylized model, which can improve the efficiency of training the second target stylized model.

[0098] Step 1003: Perform stylization processing on a plurality of third sample images using the second target stylization model to obtain a plurality of paired data.

[0099] Optionally, each third sample image may also be a face image in FFHQ and / or CelebA. Generally, each third sample image may be different from each second sample image, or may be the same as each second sample image, which is not limited in the embodiments of the present application.

[0100] In addition, the number of the third sample images may also be greater than the number of the second sample images or the number of the first sample images.

[0101] Optionally, each pairing data includes: a real face image and a corresponding stylized image of a preset style. Generally, the number of each pairing data is equal to or greater than the number of each third sample image.

[0102] Exemplarily, the real face images in each pairing data may be the face images in each third sample image input into the second target stylization model.

[0103] The stylized image of the preset style in each pairing data may refer to the stylized image output by the second target stylized model.

[0104] The real face image and the stylized image of the preset style in any paired data are one-to-one corresponding, and their semantic information is consistent.

[0105] Optionally, the preset style may refer to the above-mentioned specific style type, which may be determined by the first stylized data used in training. That is, the preset style label in each first stylized data input into the second initial stylized model may also be used to indicate the preset style of the stylized image that can be generated by the second target stylized model.

[0106] It is worth noting that the stylization processing of the plurality of third sample images by the second target stylization model may be an operation of converting each third sample image into a stylized image of the preset style. The number of each pairing data is equal to or greater than the number of each third sample image, and the second target stylization model is obtained through training, which can ensure that the quality of the stylized image output by the second target stylization model is high. In this way, a large amount of high-quality pairing data can be obtained for subsequent processing.

[0107] Step 1004: Train the third initial stylized model based on each pairing data to obtain a third target stylized model.

[0108] Optionally, the third initial stylization model may also be a generative adversarial network, the generator of the third initial stylization model is an encoder-decoder network, and the discriminator is the architecture of PatchGAN.

[0109] The third initial stylized model may be a new, untrained model, and the third initial stylized model may also be the second target stylized model, which is not limited in the embodiment of the present application.

[0110] Exemplarily, in the process of training the third initial stylized model based on each pairing data, the third initial stylized model can learn the mapping relationship between the real face image in any pairing data and the stylized image of the preset style in any pairing data, so that the third initial stylized model can be trained in a paired data manner.

[0111] The third target stylized model may be a trained third initial stylized model.

[0112] In addition, the third initial stylized model may be trained based on each processed and screened pairing data after post-processing and screening, and the embodiment of the present application is not limited to this.

[0113] It is worth noting that since each pairing data is a large amount of data generated by stylizing each third sample image through the trained second target stylization model, it is not necessary to spend a lot of time and manpower to collect a large amount of pairing data. In addition, the quality of the stylized images of the preset style in each pairing data is high, and the real face images in each pairing data are high-definition face images in FFHQ and / or CelebA, so the quality of each pairing data can be ensured to be high.

[0114] In this way, the efficiency of training the model can be improved while ensuring the performance of the trained model.

[0115] In the embodiment of the present application, a first target stylized model is obtained by training a first initial stylized model based on a plurality of first sample images, and a plurality of second sample images are subjected to random stylization processing by the first target stylized model to obtain a plurality of first stylized data. In this way, a first target stylized model can be obtained by training a small number of first sample images, and then a large number of second sample images can be subjected to random stylization processing by the first target stylized model to obtain a large number of first stylized data.

[0116] The second initial stylized model is trained based on the preset style label, each second sample image, and the stylized image in the first stylized data to obtain a second target stylized model. Since the preset style label can indicate the style type learned by the second initial stylized model, the performance of the second target stylized model in stylizing an image of a specific style type can be improved. In addition, there is no need to use a large amount of paired data to train the second target stylized model, which can improve the efficiency of training the second target stylized model.

[0117] The second target stylized model is used to perform stylized processing on the plurality of third sample images to obtain a plurality of paired data. Since the number of each paired data is equal to or greater than the number of each third sample image, and the quality of the stylized image output by the second target stylized model is relatively high, a large amount of high-quality paired data can be obtained for subsequent processing.

[0118] The third initial stylized model is trained based on each paired data to obtain a third target stylized model. Since each paired data is a large amount of data generated by stylizing each third sample image with the trained second target stylized model, it is not necessary to spend a lot of time and manpower to collect a large amount of paired data. In addition, the quality of the stylized image of the preset style in each paired data is high, and the real face image in each paired data is a high-definition face image in FFHQ and / or CelebA, so the quality of each paired data can be ensured to be high.

[0119] That is, the image stylization model training method provided in the embodiment of the present application can improve the efficiency of training the model while ensuring the performance of the trained model.

[0120] In this way, the efficiency of training the model and the performance of the trained model can be improved.

[0121] For a possible implementation, see Figure 2 , training a second initial stylized model based on a preset style label, the second sample image, and the stylized image in the first stylized data to obtain a second target stylized model, including:

[0122] Step 1005: The second initial stylization model outputs a first generation result based on the preset style label and the second sample image, and determines first loss information according to the second sample image, the first generation result and the stylized face in the first stylization data.

[0123] Optionally, the first generated result may refer to a stylized image output by the second initial stylization model during a training process.

[0124] The first loss information may be used to indicate a difference between the first generated result and the stylized face in the first stylized data.

[0125] Generally, the larger the first loss information is, the greater the difference between the first generated result and the stylized face in the first stylized data is.

[0126] In this way, the performance of the second initial stylization model can be accurately reflected according to the first loss information.

[0127] Step 1006: iteratively optimize the second initial stylized model according to the first loss information until a first preset condition is satisfied, and use the second initial stylized model that satisfies the first preset condition as the second target stylized model.

[0128] Optionally, the first preset condition may mean that the number of iterative optimizations of the second initial stylized model reaches a preset number, or the re-determined first loss information of the second initial stylized model is less than or equal to a preset threshold, which is not limited in this embodiment of the present application.

[0129] It is worth noting that in any optimization process of the second initial stylized model according to the first loss information, the parameters in the second initial stylized model can be adaptively adjusted according to the first loss information, and the first loss information of the second initial stylized model can be re-determined after adjusting the parameters until the first preset condition is met.

[0130] In this way, it can be ensured that the second initial stylized model can fully learn the content information of each second sample image and the stylized features of the stylized images in each first stylized data during the training process, thereby ensuring that the performance of the obtained second target stylized model is high.

[0131] For a possible implementation, see Figure 3 , determining first loss information according to the second sample image, the first generated result, and the stylized face in the first stylized data, including:

[0132] Step 1007: Extract feature information of the first generated result and feature information of the real face image in the second sample image based on the VGG model, and determine the content loss value based on the feature information of the first generated result and feature information of the real face image in the second sample image.

[0133] Optionally, the VGG is used to extract feature information in a VGG feature space.

[0134] The feature information of the first generated result may refer to the feature information of the first generated result in the VGG feature space. The feature information of the second sample image may refer to the feature information of the stylized face in the first stylized data in the VGG feature space.

[0135] The content loss value may refer to the L1 distance between the first generated result and the second sample image in the VGG feature space, and may be specifically used to indicate the difference in content information between the first generated result and the second sample image.

[0136] Exemplarily, the content loss value can be calculated by the following formula:

[0137]

[0138] Wherein, x1 is the second sample image, and G(x1) is the first generation result output by the second initial stylization model according to the second sample image.

[0139] Step 1008: Convert the first generated result and the stylized face in the first stylized data into a first grayscale image and a second grayscale image respectively, and extract feature information of the first grayscale image and feature information of the second grayscale image based on the VGG model.

[0140] Optionally, the first generated result and the stylized face in the first stylized data may be converted into the first grayscale image and the second grayscale image in any possible manner, which is not limited in this embodiment of the present application.

[0141] Step 1009: Determine a texture loss value based on the feature information of the first grayscale image and the feature information of the second grayscale image.

[0142] Optionally, the texture loss value may refer to an L1 distance between the Gram matrices of the first grayscale image and the second grayscale image in the VGG feature space.

[0143] The texture loss value may be specifically used to indicate the difference in texture feature information between the first generated result and the stylized face in the first stylized data.

[0144] For example, the texture loss value can be calculated by the following formula:

[0145] Wherein, y1 is the stylized face in the first stylized data, gray(·) refers to converting the image into a grayscale image, and Gram(·) refers to the Gram matrix for calculating feature information.

[0146] Step 1010: Determine a first generative adversarial loss value based on the real face images in each second sample image, the stylized face in each first stylized data, and the generator and discriminator in the second initial stylized model.

[0147] Optionally, the first generative adversarial loss value may be used to indicate the extent to which the second initial stylization model learns the stylized features of the stylized images in each first stylization data.

[0148] Exemplarily, the first generation adversarial loss value can be calculated by the following formula:

[0149] Wherein, x1 is the second sample image, y1 is the stylized face in the first stylized data, G and D are the generator and discriminator in the second initial stylized model respectively.

[0150] Step 1011: Determine the first loss information according to the content loss value, the texture loss value and the first generative adversarial loss value.

[0151] Optionally, different weights can be set for the content loss value, the texture loss value and the first generative adversarial loss value to perform weighted summation of the content loss value, the texture loss value and the first generative adversarial loss value to calculate the first loss information. The first loss information can also be calculated by average value, square difference, etc., which is not limited to the embodiments of the present application.

[0152] In this way, the first loss information used to characterize the performance of the second initial stylization model can be accurately calculated to improve the reliability of training the second initial stylization model.

[0153] For a possible implementation, see Figure 4 , performing stylization processing on a plurality of third sample images through the second target stylization model to obtain a plurality of paired data, including:

[0154] Step 1012: Input each third sample image into the second target stylization model to obtain target stylization data output by the second target stylization model.

[0155] Optionally, the target stylization data may refer to stylized images corresponding to each third sample image output after the second target stylization model performs stylization processing on each third sample image.

[0156] Optionally, the target stylized data may further include preset styles for indicating each stylized image corresponding to each third sample image.

[0157] Step 1013: Obtain each pairing data based on the mapping relationship between each third sample image and the corresponding target stylized data.

[0158] Optionally, the mapping relationship may indicate difference information between each third sample image and the corresponding target stylized data.

[0159] That is, the target stylized data is the stylized image of the preset style in each pairing data.

[0160] In this way, a large amount of high-quality paired data can be obtained accurately and efficiently, so as to subsequently train the third initial stylization model based on each paired data.

[0161] For a possible implementation, see Figure 5 , training the third initial stylized model based on each pairing data to obtain a third target stylized model, including:

[0162] Step 1014: The third initial stylization model outputs a second generation result according to the pairing data, and determines second loss information according to the second generation result.

[0163] Optionally, the second generated result may refer to a stylized image output by the third initial stylization model during a training process.

[0164] Specifically, the third initial stylized model can output the second generated result according to the real face image in the pairing data.

[0165] The second loss information may be used to indicate a difference between the second generated result and the stylized image in the paired data.

[0166] Generally, the larger the second loss information is, the greater the difference between the second generated result and the stylized face in the first stylized data is.

[0167] In this way, the performance of the third initial stylization model can be accurately reflected according to the second loss information.

[0168] Step 1015: iteratively optimize the third initial stylized model according to the second loss information until a second preset condition is satisfied, and use the third initial stylized model that satisfies the second preset condition as the third target stylized model.

[0169] Optionally, the second preset condition may mean that the number of iterative optimizations of the third initial stylized model reaches a preset number, or the re-determined second loss information of the third initial stylized model is less than or equal to a preset threshold, which is not limited in this embodiment of the present application.

[0170] It is worth noting that, in any process of optimizing the third initial stylized model according to the second loss information, the parameters in the third initial stylized model can also be adaptively adjusted according to the second loss information until the second preset condition is met.

[0171] In this way, it can be ensured that the performance of the third target stylized model is high.

[0172] For a possible implementation, see Figure 6 , determining second loss information according to the second generation result, including:

[0173] Step 1016: Determine a reconstruction loss value according to the second generated result and the target stylized data in the paired data.

[0174] Optionally, the reconstruction loss value may refer to an L1 loss between the second generated result and the target stylized data in the paired data.

[0175] The reconstruction loss value is used to indicate the mapping relationship between different features of the second generated result and the target stylized data in the paired data, for example, it can indicate the mapping relationship between the eyes of a person in the second generated result and the eyes of a person in the target stylized data in the paired data.

[0176] Exemplarily, the reconstruction loss value may be calculated by the following formula:

[0177]

[0178] Wherein, y is the target stylized data in the paired data and / or the stylized image of a preset style in the paired data, and G(x1) is the second generated result.

[0179] Step 1017: Determine a first perceptual loss value according to the feature information of the real face image in the pairing data, the feature information of each second generated result, and a preset feature parameter.

[0180] Optionally, the first perceptual loss value refers to: processing the second generated result and the real face image in the paired data to obtain a loss value for characterizing the similarity between the facial structure of the real face image in the paired data and the facial structure of the second generated result at the perceptual level. The first perceptual loss value can ignore local appearance changes, which is not limited in the embodiments of the present application.

[0181] Exemplarily, the first perceived loss value can be calculated by the following formula:

[0182]

[0183] Among them, l is the number of layers for extracting features, H l W l is the height and width of the l-th layer feature, w is the weight of the l-th layer feature, is the output result of the lth layer after the second generated result is subjected to feature extraction and normalization by the VGG network, It is the output result of the lth layer after the real face image in the paired data is extracted by the VGG network and normalized.

[0184] Optionally, the preset feature parameters may include parameters such as the number of feature layers, the height and width of the feature, and the weight of the feature.

[0185] Step 1018: Determine a second generative adversarial loss value based on each third sample image, each second generated result, each target stylized data, and the generator and discriminator in the third initial stylized model.

[0186] Optionally, each target stylized data is a stylized image of a preset style in the paired data.

[0187] Exemplarily, the second generation adversarial loss value may be calculated by referring to the formula for calculating the first generation adversarial loss value in step 1010. It should be noted that the corresponding symbols or parameters in the formula need to be replaced with the second generation result corresponding to the third initial stylized model, the third sample image (that is, the real face image in the paired data) and the target stylized data, and the embodiments of the present application are not described in detail here.

[0188] Step 1019: Determine second loss information based on the reconstruction loss value, the first perceptual loss value and the second generative adversarial loss value.

[0189] Optionally, the second loss information may be calculated by weighted sum, average value, square difference, etc., which is not limited in the embodiments of the present application.

[0190] In this way, the second loss information used to characterize the performance of the third initial stylization model can be accurately calculated to improve the reliability of training the third initial stylization model.

[0191] For a possible implementation, see Figure 7 , training a first initial stylization model based on a plurality of first sample images to obtain a first target stylization model, including:

[0192] Step 1020: performing a preprocessing operation on each first sample image, and using the preprocessed first sample image as each first sample image.

[0193] Optionally, the preprocessing operation includes at least one of the following: face alignment processing and normalization processing.

[0194] Generally, each first sample image can be taken out from FFHQ, face alignment processing can be performed on the faces in each first sample image, and then normalization processing can be performed on the first sample images after the face alignment processing.

[0195] Specifically, the Dlib library can be used to detect the face in the image, and the face area can be cropped and aligned according to the detected eye center point and mouth key point. Then, each pixel of each first sample image is divided by 255, then subtracted by 0.5 and then divided by 0.5, and normalized to the range of [-1, 1] to achieve normalization processing.

[0196] Step 1021: The first initial stylization model outputs third generation results based on the first sample images.

[0197] Optionally, the third generated result may refer to a stylized image output by the first initial stylization model during the training process.

[0198] Each third generation result corresponds to each first sample image.

[0199] Step 1022: Determine a third generative adversarial loss value based on the random latent vector, each first sample image, the generator and the discriminator in the first initial stylization model.

[0200] Optionally, the random latent vector may be a latent vector based on the Z+ space, and the random latent vector may be randomly generated, which is not limited in the embodiment of the present application.

[0201] Specifically, the third generative adversarial loss value may be calculated based on the stylized faces in each first sample image.

[0202] Exemplarily, the third generation adversarial loss value can be calculated by the following formula:

[0203] Among them, x2 is the first sample image, z is the random latent vector, G and D are the generator and discriminator in the first initial stylization model respectively, and ada is the data enhancement strategy used.

[0204] The data enhancement strategy may refer to an operation of changing the third generation result generated by the generator during the training process of the discriminator and the generator. The data enhancement strategy may include processing such as rotation, scaling, and color jitter, which is not limited in the embodiment of the present application.

[0205] Step 1023: Determine a second perceptual loss value according to the feature information of each first sample image, the feature information of each third generated result, and a preset feature parameter.

[0206] Optionally, the second perceptual loss value may be calculated by referring to the formula for calculating the first perceptual loss value in step 1017. It should be noted that the meanings of the corresponding symbols or parameters in the formula need to be replaced with the third generated result corresponding to the first initial stylized model, the real face image in each first sample image and / or the corresponding stylized face, which will not be described in detail in the embodiments of the present application.

[0207] Step 1024: Determine third loss information based on the third generated adversarial loss value and the second perceptual loss value.

[0208] Optionally, the third loss information may be used to indicate a difference between the third generation result and the stylized images in the first sample images.

[0209] Generally, the larger the third loss information is, the greater the difference between the third generation result and the stylized images in the first sample images is.

[0210] In this way, the performance of the first initial stylization model can be accurately reflected according to the third loss information.

[0211] Step 1025: iteratively optimize the first initial stylized model according to the third loss information until a third preset condition is satisfied, and use the first initial stylized model that satisfies the third preset condition as the first target stylized model.

[0212] Optionally, the third preset condition may mean that the number of iterative optimizations of the first initial stylized model reaches a preset number, or the re-determined third loss information of the first initial stylized model is less than or equal to a preset threshold, which is not limited in this embodiment of the present application.

[0213] It is worth noting that, in any process of optimizing the first initial stylized model according to the third loss information, the parameters in the first initial stylized model can also be adaptively adjusted according to the third loss information until the third preset condition is met.

[0214] In this way, it can be ensured that the performance of the first target stylized model is high.

[0215] It should be noted that the first target stylized model is obtained by training the first initial stylized model with a small amount of first sample images, and the first target stylized model outputs a large amount of first stylized data, and then the second initial stylized model is trained based on the first stylized data to obtain the second target stylized model, and a large amount of paired data is obtained from the second target stylized model. In this way, it is not necessary to spend a lot of manpower and time to collect paired data for training the third initial stylized model, and the quality of each paired data can also be ensured to be high.

[0216] In this way, the efficiency of training the model and the performance of the trained model can be improved.

[0217] The following describes the apparatus, device, and computer-readable storage medium used to execute the stylized model training method for images provided in the present application. The specific implementation process and technical effects are described above and will not be repeated below.

[0218] Figure 8 is a schematic diagram of the structure of an image stylized model training device provided in an embodiment of the present application, see Figure 8 , the device comprises:

[0219] The first execution module 201 is used to train a first initial stylization model based on multiple first sample images to obtain a first target stylization model, and perform random stylization processing on multiple second sample images through the first target stylization model to obtain multiple first stylization data.

[0220] The second execution module 202 is used to train the second initial stylized model based on the preset style label, each second sample image and the stylized image in the first stylized data to obtain a second target stylized model.

[0221] The second execution module 202 is further configured to perform stylization processing on a plurality of third sample images by using the second target stylization model to obtain a plurality of paired data.

[0222] The third execution module 203 is used to train the third initial stylization model based on each pairing data to obtain a third target stylization model.

[0223] The above-mentioned device is used to execute the method provided by the aforementioned embodiment, and its implementation principle and technical effect are similar, which will not be repeated here.

[0224] The above modules may be one or more integrated circuits configured to implement the above methods, such as one or more application specific integrated circuits (ASICs), or one or more microcontrollers, or one or more field programmable gate arrays (FPGAs). For another example, when a module is implemented in the form of a processing element scheduling program code, the processing element may be a general-purpose processor, such as a central processing unit (CPU) or other processor that can call program code. For another example, these modules may be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0225] Fig. 9 is a schematic diagram of the structure of a computer device provided in an embodiment of the present application. Fig. 9 The computer device includes: a memory 301 and a processor 302. The memory 301 stores a computer program that can be run on the processor 302. When the processor 302 executes the computer program, the steps in any of the above method embodiments are implemented.

[0226] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0227] Optionally, the present application also provides a program product, such as a computer-readable storage medium,

[0228] Includes a program that, when executed by a processor, is used to perform an embodiment of the stylized model training method for any of the above-mentioned images.

[0229] In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0230] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0231] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0232] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (English: processor) to perform some steps of the methods of various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (English: Read-Only Memory, abbreviated: ROM), random access memory (English: Random Access Memory, abbreviated: RAM), disk or optical disk and other media that can store program codes.

[0233] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0234] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for training a stylized model of an image, characterized in that: The method comprises: A first initial stylization model is trained based on a plurality of first sample images to obtain a first target stylization model, and a plurality of second sample images are randomly stylized by the first target stylization model to obtain a plurality of first stylized data; each of the first sample images includes a stylized face, each of the second sample images includes a real face image, each of the first stylized data includes a stylized image and a style label, and the number of each of the first stylized data is greater than the number of each of the first sample images; Training a second initial stylized model based on a preset style label, each of the second sample images, and the stylized image in the first stylized data to obtain a second target stylized model, wherein the preset style label is a style label in each of the first stylized data; Performing stylization processing on a plurality of third sample images by using the second target stylization model to obtain a plurality of paired data, each of the paired data comprising: a real face image and a corresponding stylized image of a preset style; The third initial stylized model is trained based on each of the paired data to obtain a third target stylized model.

2. The image stylization model training method according to claim 1, characterized in that: The step of training the second initial stylized model based on the preset style label, the second sample image, and the stylized image in the first stylized data to obtain the second target stylized model includes: The second initial stylized model outputs a first generation result based on the preset style label and the second sample image, and determines first loss information according to the second sample image, the first generation result and the stylized face in the first stylized data; The second initial stylized model is iteratively optimized according to the first loss information until a first preset condition is satisfied, and the second initial stylized model that satisfies the first preset condition is used as the second target stylized model.

3. The image stylization model training method according to claim 2, characterized in that: The determining first loss information according to the second sample image, the first generation result, and the stylized face in the first stylized data includes: Extracting feature information of the first generated result and feature information of a real face image in the second sample image based on a VGG model, and determining a content loss value based on the feature information of the first generated result and the feature information of the real face image in the second sample image; Respectively converting the first generated result and the stylized face in the first stylized data into a first grayscale image and a second grayscale image, and extracting feature information of the first grayscale image and feature information of the second grayscale image based on the VGG model; Determine a texture loss value based on the feature information of the first grayscale image and the feature information of the second grayscale image; Determine a first generative adversarial loss value based on the real face images in each of the second sample images, the stylized faces in each of the first stylized data, and the generator and the discriminator in the second initial stylized model; The first loss information is determined according to the content loss value, the texture loss value and the first generative adversarial loss value.

4. The image stylization model training method according to claim 1, characterized in that: The performing stylization processing on the plurality of third sample images by using the second target stylization model to obtain a plurality of paired data includes: Inputting each of the third sample images into the second target stylization model to obtain target stylization data output by the second target stylization model; Based on the mapping relationship between each of the third sample images and the corresponding target stylized data, each of the pairing data is obtained.

5. The image stylization model training method according to claim 1, characterized in that: The step of training the third initial stylized model based on each of the paired data to obtain a third target stylized model includes: Outputting a second generation result by the third initial stylization model according to the pairing data, and determining second loss information according to the second generation result; The third initial stylized model is iteratively optimized according to the second loss information until a second preset condition is satisfied, and the third initial stylized model that satisfies the second preset condition is used as the third target stylized model.

6. The image stylization model training method according to claim 5, characterized in that: The determining the second loss information according to the second generation result includes: Determine a reconstruction loss value according to the second generated result and the target stylized data in the paired data; Determine a first perceptual loss value according to feature information of a real face image in each of the paired data, feature information of each of the second generated results, and a preset feature parameter; Determine a second generative adversarial loss value based on each of the third sample images, each of the second generated results, each of the target stylized data, and a generator and a discriminator in the third initial stylized model; Second loss information is determined according to the reconstruction loss value, the first perceptual loss value, and the second generative adversarial loss value.

7. The image stylization model training method according to claim 1, characterized in that: The step of training the first initial stylization model based on the plurality of first sample images to obtain the first target stylization model includes: Performing a preprocessing operation on each of the first sample images, and using the preprocessed first sample images as each of the first sample images, wherein the preprocessing operation includes at least one of the following: face alignment processing and normalization processing; Outputting third generation results by the first initial stylization model based on the first sample images; Determine a third generative adversarial loss value based on the random latent vector, each of the first sample images, the generator and the discriminator in the first initial stylization model; Determine a second perceptual loss value according to the feature information of each of the first sample images, the feature information of each of the third generated results, and a preset feature parameter; Determine third loss information according to the third generative adversarial loss value and the second perceptual loss value; The first initial stylized model is iteratively optimized according to the third loss information until a third preset condition is satisfied, and the first initial stylized model that satisfies the third preset condition is used as the first target stylized model.

8. A stylized model training device for an image, characterized in that: The device comprises: A first execution module is configured to train a first initial stylization model based on a plurality of first sample images to obtain a first target stylization model, and to perform random stylization processing on a plurality of second sample images through the first target stylization model to obtain a plurality of first stylized data; each of the first sample images includes a stylized face, each of the second sample images includes a real face image, each of the first stylized data includes a stylized image and a style label, and the number of each of the first stylized data is greater than the number of each of the first sample images; a second execution module, configured to train a second initial stylized model based on a preset style label, each of the second sample images, and a stylized image in the first stylized data to obtain a second target stylized model, wherein the preset style label is a style label in each of the first stylized data; The second execution module is further used to perform stylization processing on a plurality of third sample images through the second target stylization model to obtain a plurality of paired data, each of the paired data comprising: a real face image and a corresponding stylized image of a preset style; The third execution module is used to train the third initial stylization model based on each of the pairing data to obtain a third target stylization model.

9. A computer device, characterized in that: include: A memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the computer program, the steps of the method described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 7.

Images