Face image processing method, device, computer equipment, and storage medium
By using aging model and makeup model, using VGG-16 structure and multi-convolution module layer network processing, the problem of low facial makeup migration efficiency in the prior art is solved, and the efficiency and accuracy of makeup treatment for years are achieved. .
Patent Information
- Application Number
- CN202111500426.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-12-09
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2041-12-09
AI Technical Summary
The existing facial makeup migration technology is based on a single face age and cannot effectively process makeup results at different ages. The dataset labeling takes time to achieve efficient facial makeup migration.
Two pre-trained network models are adopted: the aging model and the makeup model. Face aging and makeup are processed through feature extraction layer, encoder, residual block layer and decoding layer, and VGG-16 structure and multiple convolutional module layers are used. Achieve efficient makeup migration without manual labeling.
It realizes facial makeup treatment for different age groups, saves data set labeling costs, improves the efficiency and accuracy of facial makeup migration, and is suitable for makeup needs of multiple age groups.
Smart Images

Figure CN114283051B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image processing technology, and in particular to a facial image processing method, apparatus, computer equipment, storage medium and computer program product. Background Art
[0002] With the development of image processing technology, face image technology is becoming more and more advanced and perfect, which has brought great convenience to people's daily life.
[0003] As an important branch in the field of face image processing, facial aging technology has also been increasingly valued by researchers. Specifically, facial aging technology can usually predict a person's appearance at a certain time in the future. This technology is widely used in crime tracking and finding missing children. However, if makeup is applied to an aging face, it will greatly affect the resolution of the tracking system, thereby affecting the system's prediction performance of a person's future facial appearance. Makeup is a common method of using special cosmetics to improve facial appearance, such as foundation and eyeliner to hide facial blemishes. There are thousands of cosmetics that differ in brand, color, and usage. Without professional advice, it is difficult to find a suitable makeup style.
[0004] Facial makeup transfer provides an effective way to help users choose the most suitable makeup style. Existing facial makeup transfer mainly falls into two categories: traditional image processing methods, such as image gradient editing and physics-based operations, and deep learning-based methods based on deep neural networks. However, existing facial makeup transfer techniques are based on a single face age, but in fact, the results of facial makeup transfer at different ages are different. Therefore, conventional methods can only generate datasets by annotation. Such large-scale datasets are very time-consuming and cannot achieve efficient facial makeup transfer. Summary of the invention
[0005] Based on this, it is necessary to provide a facial image processing method, device, computer equipment, storage medium and computer program product that can efficiently realize facial makeup migration in response to the above technical problems.
[0006] In a first aspect, the present application provides a method for processing a face image. The method comprises:
[0007] Acquire an original image and a makeup target image corresponding to the original image;
[0008] Inputting the original image into a preset trained first network model to obtain a color image, wherein the first network model includes an aging model;
[0009] The color image and the makeup target image are input into a preset trained second network model to obtain a color image after facial makeup transfer, wherein the second network model includes a makeup model.
[0010] In one embodiment, the first network model includes a feature extraction layer, an encoder, a residual block layer, a decoding layer, and an evaluator;
[0011] The feature extraction layer adopts the network structure of VGG-16, and the output of the feature extraction layer is used as the input of the evaluator in the first network model; the residual block layer includes a convolution layer, a regularization layer and an activation function connected in sequence; the encoder and the decoding layer respectively include multiple different convolution module layers.
[0012] In one embodiment, inputting the original image into a preset trained first network model to obtain a color image comprises:
[0013] The feature extraction layer extracts the feature map of the original image, the encoding layer encodes the feature map and sends it to the residual block layer, the residual block layer maintains the size of the feature map unchanged, and transmits it to the decoding layer, and the decoding layer performs an up-convolution operation on the received feature map to obtain a color image.
[0014] In one embodiment, the training process of the first network model includes:
[0015] Get the original data set;
[0016] Selecting any one picture from the original data set to obtain a sample picture;
[0017] Inputting the sample image into an initial first network model to obtain a sample color image set, wherein the sample color image set includes a plurality of sample color images after facial aging;
[0018] Input each sample color picture in the sample color picture set into the evaluator of the initial first network model to obtain an aging result corresponding to the sample color picture, and obtain the loss of the evaluator and the loss of the generator in the initial first network model according to the aging result;
[0019] Inputting each sample color picture in the sample color picture set into the initial first network model to obtain a secondary sample color picture set, wherein the secondary sample color picture set includes a plurality of sample color pictures after secondary facial aging;
[0020] Obtaining a real age label corresponding to the sample image, and obtaining a perceptual loss function in an initial first network model according to the sample color image after secondary facial aging and the real age label;
[0021] Inputting the sample color picture set into the encoder in the initial first network model to obtain an encoding result, subjecting the encoding result to residual processing and decoding processing to obtain a facial aging result, and obtaining a texture loss decoded in the initial first network model according to the facial aging result;
[0022] The initial first network model is iteratively trained according to the loss of the evaluator and the loss of the generator, the perceptual loss, and the texture loss until the training objective optimization function of the trained first network model converges, thereby obtaining a trained first network model.
[0023] In one embodiment, the training process of the second network model includes:
[0024] Inputting the sample picture and the sample color picture set into the generator of the initial second network model to obtain a target picture set and a generated data set, and obtaining a perceptual loss and a local histogram loss according to the target picture set and the generated data set;
[0025] Inputting each target image in the target image set into the first discriminator in the initial second network model to obtain a first recognition result corresponding to each color image, and obtaining the loss of the first discriminator in the initial second network model according to the first recognition result corresponding to each color image;
[0026] Inputting each target image in the target image set into the second discriminator in the initial second network model to obtain a second recognition result corresponding to each color image, and obtaining the loss of the second discriminator in the initial second network model according to the second recognition result corresponding to each color image;
[0027] Inputting each target image in the target image set into the generator in the initial second network model to obtain a restored generated image set and a generated makeup image set, and obtaining a cycle consistency loss according to the generated image set and the makeup image set;
[0028] According to the perceptual loss, the local histogram loss, the loss of the first discriminator, the loss of the second discriminator and the cycle consistency loss, the initial second network model is iteratively trained until the training objective optimization function in the second network model converges to obtain a trained second network model.
[0029] In one of the embodiments, the evaluator in the first network model includes multiple discriminators, and the evaluation value output by the evaluator is obtained by concatenating the output results of the multiple discriminators.
[0030] In a second aspect, the present application also provides a facial image processing device. The device comprises:
[0031] A data acquisition module, used to acquire an original image and a makeup target image corresponding to the original image;
[0032] A first processing module, configured to input the original image into a preset trained first network model to obtain a color image, wherein the first network model includes an aging model;
[0033] The second processing module is used to input the color image and the makeup target image into a preset trained second network model to obtain a color image after facial makeup transfer, wherein the second network model includes a makeup model.
[0034] In a third aspect, the present application further provides a computer device. The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:
[0035] Acquire an original image and a makeup target image corresponding to the original image;
[0036] Inputting the original image into a preset trained first network model to obtain a color image, wherein the first network model includes an aging model;
[0037] The color image and the makeup target image are input into a preset trained second network model to obtain a color image after facial makeup transfer, wherein the second network model includes a makeup model.
[0038] In a fourth aspect, the present application further provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the following steps are implemented:
[0039] Acquire an original image and a makeup target image corresponding to the original image;
[0040] Inputting the original image into a preset trained first network model to obtain a color image, wherein the first network model includes an aging model;
[0041] The color image and the makeup target image are input into a preset trained second network model to obtain a color image after facial makeup transfer, wherein the second network model includes a makeup model.
[0042] In a fifth aspect, the present application further provides a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, the following steps are implemented:
[0043] Acquire an original image and a makeup target image corresponding to the original image;
[0044] Inputting the original image into a preset trained first network model to obtain a color image, wherein the first network model includes an aging model;
[0045] The color image and the makeup target image are input into a preset trained second network model to obtain a color image after facial makeup transfer, wherein the second network model includes a makeup model.
[0046] The above-mentioned facial image processing method, device, computer equipment, storage medium and computer program product obtain an original image and a makeup target image corresponding to the original image; input the original image into a preset trained first network model to obtain a color image; input the color image and the makeup target image into a preset trained second network model to obtain a color image after facial makeup migration. In the whole process, the first network model (aging model) and the second network model (makeup model) are used to obtain the color image and the color image after facial makeup migration, respectively, without relying on manual annotation, and efficient facial makeup migration can be achieved. BRIEF DESCRIPTION OF THE DRAWINGS
[0047] Figure 1 1 is a flow chart of a method for processing a face image in one embodiment;
[0048] Figure 2 The architecture diagram of the feature extraction layer, encoder, residual block layer and decoding layer in the first network model;
[0049] Figure 3 is a schematic diagram of the architecture of the convolutional module layer included in the first network model;
[0050] Figure 4 is an architecture diagram of an upper convolutional module layer included in the first network model;
[0051] Figure 5 A schematic diagram of a first network model training process in an embodiment;
[0052] Figure 6 A schematic diagram of the second network model training process in one embodiment;
[0053] Figure 7 is a structural block diagram of a face image processing device in one embodiment;
[0054] Figure 8 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION
[0055] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0056] In one embodiment, Figure 1 As shown, a face image processing method is provided, comprising the following steps:
[0057] S200: Acquire an original image and a makeup target image corresponding to the original image.
[0058] The original image refers to the original portrait image without makeup, such as the face image of user A without makeup. The makeup target image corresponding to the original image refers to the portrait image after makeup, such as the face image of user A after makeup. Specifically, the size of the original image and the makeup target image is the same, which can be 256*256*3.
[0059] S400: Inputting the original image into a preset trained first network model to obtain a color image, wherein the first network model includes an aging model.
[0060] The first network model is an aging model (Aging-Net). The trained first network model is obtained by training the initial first network model using sample data. The original image is input into the preset trained first network model to obtain a color image.
[0061] Specifically, the first network model includes a feature extraction layer, an encoder, a residual block layer, and a decoding layer PatchDecoder connected in sequence, and also includes an estimator; wherein the feature extraction layer adopts the network structure of VGG-16, and the output of the feature extraction layer serves as the input of the estimator in the first network model; the residual block layer includes a convolutional layer, a regularization layer, and an activation function connected in sequence; the encoder and the decoding layer each include a plurality of different convolutional module layers. Furthermore, the feature extraction layer retains the outputs of the 2nd, 4th, 7th, and 10th convolutional layers in the network structure of VGG-16 as the input of the estimator. The residual block layer, encoder, and decoding layer all contain a plurality of different convolutional module layers, and their specific structures will be further described in the subsequent content.
[0062] S600: Inputting the color image and the makeup target image into a preset trained second network model to obtain a color image after facial makeup transfer, wherein the second network model includes a makeup model.
[0063] The second network model is the makeup model (Makeup-Net). The process of obtaining the trained second network model is similar to that of obtaining the trained first network model. The two models can be trained together using the same training data. Specifically, the second network model includes a generator G, a first discriminator D, and a generator G connected in sequence. A and the second discriminator D B , the first discriminator D A and the second discriminator D B The network structure is the same. Furthermore, the generator G in the second network model is used twice. The first time, the generator G makes up the original image and removes the target makeup image. The second time, the generator G reverses the result of the first time and is used to calculate the cycle consistency loss.
[0064] The above-mentioned facial image processing method obtains an original image and a makeup target image corresponding to the original image; inputs the original image into a preset trained first network model to obtain a color image; inputs the color image and the makeup target image into a preset trained second network model to obtain a color image after facial makeup transfer. In the whole process, the first network model and the second network model are used to obtain the color image and the color image after facial makeup transfer, respectively, without relying on manual annotation, and efficient facial makeup transfer can be achieved.
[0065] In one embodiment, the first network model includes a feature extraction layer, an encoder, a residual block layer, a decoding layer, and an evaluator;
[0066] The feature extraction layer adopts the network structure of VGG-16, and the output of the feature extraction layer is used as the input of the evaluator in the first network model; the residual block layer includes a convolutional layer, a regularization layer and an activation function connected in sequence; the encoder and decoding layers respectively include multiple different convolutional module layers.
[0067] To illustrate the specific structure of the first network model in detail, the following will be described in detail. It should be pointed out that the quantization parameters such as input data size, convolution kernel size, step size, etc. described in the following embodiments can be adjusted accordingly according to the needs of the actual situation.
[0068] In a specific application example, the first network model (ie, the first network model Aging-Net) includes a feature extraction layer, an encoder Encoder, a residual block layer, and a decoding layer PatchDecoder (eg, Figure 2 Specifically, the feature extraction layer adopts the network structure of VGG-16, and retains the outputs of the 2nd, 4th, 7th, and 10th convolutional layers in the network structure of VGG-16 as the input of the estimator.
[0069] The residual block layer is formed by connecting the convolution layer, the regularization layer, and the activation function in series. The size of the convolution kernel is 3*3, the convolution step is 1, and the activation function uses ReLU. The structures of the encoder, residual block layer, and decoding layer PatchDecoder are as follows:
[0070] The first layer is the convolutional module layer (such as Figure 3 As shown in the figure, it consists of a convolutional layer, a regularization operation, and an activation function, with an input size of 3*224*224. The convolution kernel size is 7*7, the step size is 1, the output size is 32*224*224, and the image padding is 3. The regularization operation is instance regularization (InstanceNorm), and a learnable affine transformation parameter is added to the layer. The activation function uses ReLU.
[0071] The second layer is the convolution module layer, which consists of a convolution layer, a regularization operation, and an activation function. The input size is 32*224*224. The convolution kernel size is 3*3, the step size is 2, the image padding is 1, and the output size is 64*112*112. The regularization operation is instance regularization (InstanceNorm), and learnable affine transformation parameters are added to this layer. The activation function uses ReLU.
[0072] The third layer is the convolution module layer, which consists of a convolution layer, a regularization operation, and an activation function. The input size is 64*112*112. The convolution kernel size is 3*3, the step size is 2, the image padding is 1, and the output size is 128*56*56. The regularization operation is instance regularization (InstanceNorm), and a learnable affine transformation parameter is added to the layer. The activation function uses ReLU.
[0073] The fourth layer is the residual block layer, which consists of a convolution layer, a regularization operation, and an activation function, with an input size of 128*56*56. The convolution kernel size is 3*3, the step size is 1, the image padding is 1, and the output size is 128*56*56. The regularization operation is instance regularization (InstanceNorm), and a learnable affine transformation parameter is added to this layer. The activation function uses ReLU.
[0074] The fifth layer is the residual block layer, which consists of a convolution layer, a regularization operation, and an activation function, with an input size of 128*56*56. The convolution kernel size is 3*3, the step size is 1, the image padding is 1, and the output size is 128*56*56. The regularization operation is instance regularization (InstanceNorm), and learnable affine transformation parameters are added to this layer. The activation function uses ReLU.
[0075] The sixth layer is the residual block layer, which consists of a convolution layer, a regularization operation, and an activation function, with an input size of 128*56*56. The convolution kernel size is 3*3, the step size is 1, the image padding is 1, and the output size is 128*56*56. The regularization operation is instance regularization (InstanceNorm), and learnable affine transformation parameters are added to this layer. The activation function uses ReLU.
[0076] The seventh layer is the upper convolution module layer (such as Figure 4 As shown in the figure, it is formed by the transposed convolution layer, the regularization operation and the activation function connected in series in sequence, and the input size is 128*56*56. Among them, the convolution kernel size is 3*3, the step size is 2, the image padding is 1, the output_padding is 1, and the output size is 64*112*112. The regularization operation is instance regularization (InstanceNorm), and a learnable affine transformation parameter is added to the layer. The activation function uses ReLU.
[0077] The eighth layer is the upper convolution module layer, which is formed by the transposed convolution layer, the regularization operation, and the activation function connected in series in sequence, with an input size of 64*112*112. Among them, the convolution kernel size is 3*3, the step size is 2, the image padding is 1, the output_padding is 1, and the output size is 32*224*224. The regularization operation is instance regularization (InstanceNorm), and a learnable affine transformation parameter is added to this layer. The activation function uses ReLU.
[0078] The ninth layer is the upper convolution module layer, which is formed by the transposed convolution layer and the activation function connected in sequence, with an input size of 32*224*224. Among them, the convolution kernel size is 7*7, the step size is 1, the image padding is 3, the activation function uses Tanh, and the output size is 3*224*224.
[0079] The estimator in the first network model consists of four discriminators, and the value of Estimator is the concatenation of the output results of the four discriminators. The structure of the first discriminator of the estimator in the first network model is:
[0080] The first layer is the convolution module layer, which consists of a convolution layer, a regularization operation, and an activation function. The input size is 512*28*28. The convolution kernel size is 4*4, the step size is 2, the image padding is 1, and the output size is 512*14*14. The regularization operation is batch normalization (BatchNorm), and learnable affine transformation parameters are added to this layer. The activation function uses LeakyReLU.
[0081] The second layer is the convolution module layer, which consists of a convolution layer, a regularization operation, and an activation function, with an input size of 512*14*14. The convolution kernel size is 4*4, the step size is 2, the image padding is 1, and the output size is 512*7*7. The regularization operation is batch normalization (BatchNorm), and learnable affine transformation parameters are added to this layer. The activation function uses LeakyReLU.
[0082] The third layer is the convolution module layer, which is composed of convolution layers with an input size of 512*7*7. The convolution kernel size is 4*4, the step size is 2, the image padding is 1, and the output size is 1*3*3.
[0083] The second discriminator structure of the estimator in the first network model is as follows:
[0084] The first layer is the convolution module layer, which consists of a convolution layer, a regularization operation, and an activation function. The input size is 256*56*56. The convolution kernel size is 4*4, the step size is 2, the image padding is 1, and the output size is 512*28*28. The regularization operation is batch normalization (BatchNorm), and a learnable affine transformation parameter is added to the layer. The activation function uses LeakyReLU.
[0085] The second layer is the convolution module layer, which consists of a convolution layer, a regularization operation, and an activation function. The input size is 512*28*28. The convolution kernel size is 4*4, the step size is 2, the image padding is 1, and the output size is 512*14*14. The regularization operation is batch normalization (BatchNorm), and learnable affine transformation parameters are added to this layer. The activation function uses LeakyReLU.
[0086] The third layer is the convolution module layer, which consists of a convolution layer, a regularization operation, and an activation function, with an input size of 512*14*14. The convolution kernel size is 4*4, the step size is 2, the image padding is 1, and the output size is 512*7*7. The regularization operation is batch normalization (BatchNorm), and learnable affine transformation parameters are added to this layer. The activation function uses LeakyReLU.
[0087] The fourth layer is the convolution module layer, which is composed of convolution layers with an input size of 512*7*7. The convolution kernel size is 4*4, the step size is 2, the image padding is 1, and the output size is 1*3*3.
[0088] The third discriminator structure of the estimator in the first network model is as follows:
[0089] The first layer is the convolution module layer, which consists of a convolution layer, a regularization operation, and an activation function. The input size is 128*112*112. The convolution kernel size is 4*4, the step size is 2, the image padding is 1, and the output size is 256*56*56. The regularization operation is batch normalization (BatchNorm), and learnable affine transformation parameters are added to this layer. The activation function uses LeakyReLU.
[0090] The second layer is the convolution module layer, which consists of a convolution layer, a regularization operation, and an activation function. The input size is 256*56*56. The convolution kernel size is 4*4, the step size is 2, the image padding is 1, and the output size is 512*28*28. The regularization operation is batch normalization (BatchNorm), and learnable affine transformation parameters are added to this layer. The activation function uses LeakyReLU.
[0091] The third layer is the convolution module layer, which consists of a convolution layer, a regularization operation, and an activation function, with an input size of 512*28*28. The convolution kernel size is 4*4, the step size is 2, the image padding is 1, and the output size is 512*14*14. The regularization operation is batch normalization (BatchNorm), and learnable affine transformation parameters are added to this layer. The activation function uses LeakyReLU.
[0092] The fourth layer is the convolution module layer, which consists of a convolution layer, a regularization operation, and an activation function, with an input size of 512*14*14. The convolution kernel size is 4*4, the step size is 2, the image padding is 1, and the output size is 512*7*7. The regularization operation is batch normalization (BatchNorm), and learnable affine transformation parameters are added to this layer. The activation function uses LeakyReLU.
[0093] The fifth layer is the convolution module layer, which is composed of convolution layers with an input size of 512*7*7. The convolution kernel size is 4*4, the step size is 2, the image padding is 1, and the output size is 1*3*3.
[0094] The fourth discriminator structure of the estimator in the first network model is as follows:
[0095] The first layer is the convolution module layer, which consists of a convolution layer, a regularization operation, and an activation function. The input size is 64*224*224. The convolution kernel size is 4*4, the step size is 2, the image padding is 1, and the output size is 128*112*112. The regularization operation is batch normalization (BatchNorm), and learnable affine transformation parameters are added to this layer. The activation function uses LeakyReLU.
[0096] The second layer is the convolution module layer, which consists of a convolution layer, a regularization operation, and an activation function. The input size is 128*112*112. The convolution kernel size is 4*4, the step size is 2, the image padding is 1, and the output size is 256*56*56. The regularization operation is batch normalization (BatchNorm), and learnable affine transformation parameters are added to this layer. The activation function uses LeakyReLU.
[0097] The third layer is the convolution module layer, which consists of a convolution layer, a regularization operation, and an activation function. The input size is 256*56*56. The convolution kernel size is 4*4, the step size is 2, the image padding is 1, and the output size is 512*28*28. The regularization operation is batch normalization (BatchNorm), and learnable affine transformation parameters are added to this layer. The activation function uses LeakyReLU.
[0098] The fourth layer is the convolution module layer, which consists of a convolution layer, a regularization operation, and an activation function, with an input size of 512*28*28. The convolution kernel size is 4*4, the step size is 2, the image padding is 1, and the output size is 512*14*14. The regularization operation is batch normalization (BatchNorm), and learnable affine transformation parameters are added to this layer. The activation function uses LeakyReLU.
[0099] The fifth layer is the convolution module layer, which consists of a convolution layer, a regularization operation, and an activation function, with an input size of 512*14*14. The convolution kernel size is 4*4, the step size is 2, the image padding is 1, and the output size is 512*7*7. The regularization operation is batch normalization (BatchNorm), and learnable affine transformation parameters are added to this layer. The activation function uses LeakyReLU.
[0100] The sixth layer is the convolution module layer, which is composed of convolution layers, with an input size of 512*7*7. The convolution kernel size is 4*4, the step size is 2, the image padding is 1, and the output size is 1*3*3. Finally, the outputs of the four discriminators are connected, and the authenticity of the image is reduced to the minimum possible through iterative mean square error loss.
[0101] In one embodiment, the original image is input into a preset trained first network model to obtain a color image, including: the feature extraction layer extracts the feature map of the original image, the encoding layer encodes the feature map and sends it to the residual block layer, the residual block layer maintains the size of the feature map unchanged, and transmits it to the decoding layer, and the decoding layer performs an up-convolution operation on the received feature map to obtain a color image.
[0102] Specifically, the trained first network model includes a feature extraction layer, a coding layer, a residual block layer, and a decoding layer. The input original picture is processed in the above order, and finally a color picture is obtained. In actual applications, the original picture is input into the first network model, the feature extraction layer extracts the feature map of the original picture, the feature extraction layer sends the feature map to the coding layer, and the coding layer obtains a feature map of size 128*56*56. After encoding the data, it is sent to 4 residual blocks. The size of the feature map remains unchanged, and then the obtained feature map is transmitted to the decoding layer (PatchDecoder), and the feature map is subjected to an up-convolution operation, and a color picture of size 224*224*3 is obtained.
[0103] like Figure 5 As shown, in one embodiment, the training process of the first network model includes:
[0104] S310: Obtaining an original data set;
[0105] S320: randomly select a picture from the original data set to obtain a sample picture;
[0106] S330: Inputting the sample image into the initial first network model to obtain a sample color image set, wherein the sample color image set includes a plurality of sample color images after facial aging;
[0107] S340: Input each sample color picture in the sample color picture set to the evaluator of the initial first network model to obtain an aging result corresponding to the sample color picture, and obtain the loss of the evaluator and the loss of the generator in the initial first network model according to the aging result;
[0108] S350: inputting each sample color picture in the sample color picture set into the initial first network model to obtain a secondary sample color picture set, wherein the secondary sample color picture set includes a plurality of sample color pictures after secondary facial aging;
[0109] S360: Obtain a real age label corresponding to the sample image, and obtain a perceptual loss function in the initial first network model according to the sample color image after secondary facial aging and the real age label;
[0110] S370: Inputting the sample color picture set into the encoder in the initial first network model to obtain an encoding result, subjecting the encoding result to residual processing and decoding processing to obtain a facial aging result, and obtaining a texture loss decoded in the initial first network model according to the facial aging result;
[0111] S380: Iteratively train the initial first network model according to the loss of the evaluator and the loss of the generator, the perceptual loss, and the texture loss until the training objective optimization function of the trained first network model converges, thereby obtaining the trained first network model.
[0112] In order to explain the training process of the first network model in detail, a specific application example will be used below, and a detailed and complete mathematical formula will be used for description.
[0113] (1-1) From the original data set (I (1) ,I (2) ,...,I (N) ) Choose any picture I (i) , put the picture I (i) Input into the PatchAutoEncoder of the first network model to obtain a set of 224*224*3 facial aging color images (I src(1) ,I src(2) ,...,I src(N) ), where N represents the total number of images in the original dataset, and i∈[1,N].
[0114] (1-2) The set (I) obtained in step (1-1) src(1) ,I src(2) ,...,I src(N) ) is input into the estimator to obtain the aging results corresponding to all color images (I' src(1) ,I' src(2) ,...,I' src(N) ), according to the aging results corresponding to all color images (I' src(1) ,I' src(2) ,...,I' src(N) )Get the loss of the estimator and the loss of the generator;
[0115] The loss of the estimator is equal to:
[0116]
[0117] Where x represents the original data set (I src(1) ,I src(2) ,...,I src(N) ), G(x) represents the first facial aging result (I' src(1) ,I' src(2) ,...,I' src(N) ) in the xth data.
[0118] The loss of the generator is equal to:
[0119]
[0120] Where x represents the original data set (I src(1) ,I src(2) ,...,I src(N) ), G(x) represents the first facial aging result (I' src(1) ,I' src(2) ,...,I' src(N) ) in the xth data.
[0121] (1-3) The set (I) obtained in step (1-1) src(1) ,I src(2) ,...,I src(N) ) is input into Patch AutoEncoder to obtain the facial aging results corresponding to all color images (I' src(1) ,I' src(2) ,...,I' src(N) ), according to the facial aging results (I' src(1) ,I' src(2) ,...,I' src(N) ) and the true age label y(1)',y(2)',...,y(N)') to obtain the perceptual loss;
[0122]
[0123] Where x represents the original data set (I src(1) ,I src(2) ,...,I src(N) ), G(x) represents the first facial aging result (I' src(1) ,I' src(2) ,...,I' src(N) ) in the xth data.
[0124] (1-4) The set (I) obtained in step (1-1) src(1) ,I src(2) ,...,I src(N) ) are input into the encoder to obtain the encoding result (I E(1) ,I E(2) ,...,I E(N) ), the encoding result is passed through 4 residual blocks to obtain the feature map (I' E(1) ,I' E(2) ,...,I' E(N) ), the obtained feature map (I' E(1) ,I' E(2) ,...,I' E(N)) is sent to PatchDecoder and the facial aging result is obtained (I' src(1) ,I' src(2) ,...,I' src(N) )Get the texture loss of PatchDecoder;
[0125] Among them, the texture loss of PatchDecoder is equal to:
[0126]
[0127] where w l is a hyperparameter, which indicates the weight value contributed by layer l to the total loss. l is the feature correlation between the original image x and the generated image x', given by the Gram matrix at a constant ratio:
[0128]
[0129] in, represents the Gram matrix, G l ∈R Nl×Nl , l represents each layer in the network, G l It is a description of the features of each layer in a given texture network, which corresponds to the texture features in our network. For a given original image x', we first calculate the activation of each layer l in the convolutional neural network. This feature acts on each layer of image generation and is composed of the correlation between each layer. Its feature correlation function as follows:
[0130]
[0131] (1-5) According to the loss of the evaluator and the loss of the generator obtained in step (1-2), the perceptual loss obtained in step (1-3), and the texture loss obtained in step (1-4), the first network model is iteratively trained until the training objective optimization function of the first network model converges.
[0132] Furthermore, the training objective optimization function of the first network model is:
[0133] (a) Optimization function of the discriminator:
[0134] L D =L adv (D)
[0135] (b) The optimization function of the generator:
[0136] L G =λ adv L adv (G)+λid L identity +λ tex L texure
[0137] Among them, λ adv =50,λ id =0.5,λ tex =50, the discriminator and generator are trained alternately until the model converges.
[0138] like Figure 6 As shown, in one embodiment, the training process of the second network model includes:
[0139] S510: Inputting the sample picture and the sample color picture set into the generator of the initial second network model to obtain the target picture set and the generated data set, and obtaining the perceptual loss and the local histogram loss according to the target picture set and the generated data set;
[0140] S520: input each target image in the target image set into the first discriminator in the initial second network model to obtain a first recognition result corresponding to each color image, and obtain the loss of the first discriminator in the initial second network model according to the first recognition result corresponding to each color image;
[0141] S530: inputting each target image in the target image set into the second discriminator in the initial second network model to obtain a second recognition result corresponding to each color image, and obtaining a loss of the second discriminator in the initial second network model according to the second recognition result corresponding to each color image;
[0142] S540: inputting each target image in the target image set into the generator in the initial second network model to obtain a restored generated image set and a generated makeup image set, and obtaining a cycle consistency loss according to the generated image set and the makeup image set;
[0143] S550: Iteratively train the initial second network model according to the perceptual loss, the local histogram loss, the loss of the first discriminator, the loss of the second discriminator, and the cycle consistency loss until the training objective optimization function in the second network model converges to obtain a trained second network model.
[0144] In order to explain the training process of the second network model in detail, a specific application example will be used below, and a detailed and complete mathematical formula will be used for description.
[0145] (1-6) successively convert the set (I) obtained in step (1-1) src(1) ,I src(2) ,...,I src(N) ) and the original makeup image set (Itarget(1) ,I target(2) ,...,I target(N) ) is input into the generator G of the second network model to obtain the target image set (I' target(1) ,I' target(2) ,...,I' target(N) ) and the generated data set (I' src(1) ,I' src(2) ,...,I' src(N) ), we can get the perceptual loss L perceptual and local histogram loss L makeup :
[0146] The perceptual loss is:
[0147]
[0148]
[0149] H l , W l and C l Respectively represent the height, width and number of feature maps, F l i,j Represents the l-th layer feature information extracted from the VGG network.
[0150] Among them, the local histogram loss is:
[0151] L makeup =λ f L face +λ l L lips +λ e L eyes
[0152] L i =||I′ src (x)-HM(I′ src (x) FP i (I′ src (x)),I tgt (t) FP i (I tgt (t)))||2
[0153] Where i∈{face,lips,eyes},λ f , l , e is the weight coefficient of each calculation area.
[0154] (1-7) The target image set (I') obtained in step (1-6) target(1) ,I'target(2) ,...,I' target(N) ) is input into the discriminator D of the second network model. A In order to obtain the recognition results corresponding to all color images (I" target(1) ,I” target(2) ,...,I” target(N) ), according to the recognition results corresponding to all color images (I" target(1) ,I” target(2) ,...,I” target(N) ) Get the discriminator D A Loss:
[0155]
[0156] (1-8) The target image set (I') obtained in step (1-6) src(1) ,I' src(2) ,...,I' src(N) ) is input into the discriminator D of the second network model. B In order to obtain the recognition results corresponding to all color images (I" src(1) ,I” src(2) ,...,I” src(N) ), according to the recognition results corresponding to all color images (I" src(1) ,I” src(2) ,...,I” src(N) ) Get the discriminator D B loss;
[0157]
[0158] (1-9) The target image set (I') obtained in step (1-6) target(1) ,I' target(2) ,...,I' target(N) ) and the generated data set (I' src(1) ,I' src(2) ,...,I' src(N) ) is input into the generator G of the second network model to obtain the restored generated picture set And the generated makeup pictures Set to get the cycle consistency loss L cycle :
[0159]
[0160] (1-10) According to step (1-6), the perceptual loss L is obtained perceptual and local histogram loss L makeup , the discriminator D obtained in step (1-7)A The loss of L adv (D A ), the discriminator D obtained in step (1-8) B The loss of L adv (D B ) and the cycle consistency loss L obtained in step (1-9) cycle The second network model is iteratively trained until the training objective optimization function of the second network model converges.
[0161] Furthermore, in the iteration process, an iteration stop condition may be set to obtain the trained first network model and the trained second network model. The iteration stop condition is specifically as follows:
[0162] (1-11) Repeat the above steps (1-1) to (1-10) until the number of iterations reaches the maximum set number of iterations, or the prediction error of the artificial intelligence model to be tested is greater than 0.5, or the loss function of the alternative model is greater than 0.5, thereby obtaining the trained first network model and the second network model.
[0163] In addition, the training optimization function of the second network model is:
[0164] (c) Optimization function of the discriminator:
[0165] L adv (D) = L adv (D A )+L adv (D B )
[0166] (d) The optimization function of the generator:
[0167] L adv (G) = λ adv L adv +λ per L perceptual +λ cycle L cycle +λ m L makeup
[0168] Among them, λ adv =5,λ per =0.05,λ cycle =10,λ m =1,λ f =λ l =λ e = 2, the discriminator and generator are trained alternately until the model converges.
[0169] Furthermore, the training process may also include, after the above step (1-11), using the test data set to perform facial aging and makeup processing on the trained first network model and the second network model, and counting the aging results and facial makeup results.
[0170] Specifically, this step is to judge each aging result and facial makeup result in the test, so as to obtain the aging and facial makeup performance of the system. 200 volunteers are randomly selected. The results after the test are judged good or bad, and each person judges each test result.
[0171] Preferably, the aging datasets used in the present invention are CACD and Morph-II, and the facial makeup dataset is Makeup-dataset. CACD has 160,000 pictures, 90% of which are used for training and the rest for testing. The aging intervals are {0-20, 21-30, 31-40, 41-50, 50+.}; Morph-II has 52,099 pictures, which are mainly used for performance testing. 90% of the data in Makeup-dataset is used for training and the rest is used for testing.
[0172] Preferably, the method of the present invention is implemented using the pytorch framework and implemented on NVIDIA Tesla P100, with a batch size of 4 and a maximum number of iterations of 200000. The training method uses a 10-fold cross validation method, and the Adam optimizer is used to update the system.
[0173] In order to verify the effect of the face image processing method of the present invention, the following will be explained based on experimental data.
[0174] All experiments in this paper are implemented on NVIDIA Tesla P100 using pytorch, with a batch size of 4, a maximum number of iterations of 200,000, and a 10-fold cross-validation method. The entire system training took 96 hours. In this experiment, the Adam optimizer was used, with an initial learning rate of 0.0002, and
[0175] The comparison algorithms are the best algorithms CAAE network and a pyramid architecture of GANs network, and the age ranges are {0-20, 21-30, 31-40, 41-50, 50+}. From the experimental results, we can see that our algorithm can learn the aging characteristics of the face well and can display them realistically.
[0176] Our algorithm can realistically learn the facial makeup characteristics of different ages and display them realistically.
[0177] Under the same test data set, the performance of the present invention is compared with the latest facial makeup algorithms Face makeup by example, DMT, and CycleGAN, and the comparison is made from the following three aspects: (1) facial makeup quality, that is, whether the generated image successfully applies the reference makeup style; (2) facial identity accuracy, that is, whether the final generated image is still the original person; (3) image quality, that is, whether the image is clear; the performance of the facial aging algorithm of the present invention is compared with the baseline model CAAE network and a pyramid architecture of GANs network in terms of the performance of each model in learning facial aging features. Table 1 gives the detailed results, from which it can be concluded that our proposed method performs best in these three aspects of facial makeup, and also performs best in facial aging performance.
[0178] The test set was aged using the face image processing algorithm of the applicant, and the API provided by the third-party agency Face++ was used to identify the age range of the original data and the generated faces, and the qualified aging results were counted. Table 2 shows the prediction results obtained by the third-party agency Face++. As can be seen from Table 2, most of the results predicted by Face++ fall within the specified age range, which also proves the effectiveness of our algorithm.
[0179] Table 1 Comparison of aging / makeup results of different algorithms
[0180]
[0181]
[0182] Table 2 Prediction accuracy of Face++ for face aging processing results
[0183]
[0184] In the above embodiments, the face image processing method of the present application has the following significant technical effects:
[0185] (1) Since the present application first performs facial aging learning for different age groups and then obtains corresponding facial makeup results based on the facial aging characteristics of different age groups, it can solve the technical problem that the existing facial makeup algorithm has poor performance because it is based on facial makeup of the same person at a single age group.
[0186] (2) Since the present application adopts steps (1-5) and (1-10), it ensures that the original facial makeup dataset of the same person in a single age group learns realistic facial aging features and maintains the original characteristics of the facial image. Therefore, it is possible to generate a large-scale facial makeup dataset of different age groups and save a lot of overhead required for collecting and annotating the dataset.
[0187] (3) Since the present application adopts steps (1-6), (1-7) and (1-8), any picture can be made up to apply makeup to the face of people of multiple age groups, which can greatly meet the needs of real applications.
[0188] (4) Since the present application uses the first network model and the second network model for end-to-end training, this ensures that the entire system is trained and converged faster, thereby generating facial makeup migration images more efficiently.
[0189] (5) Since the network structure used in this application is simple, the model can be easily deployed and takes a short time to run, making it ideal for robustness testing of some black box models in reality.
[0190] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.
[0191] Based on the same inventive concept, the embodiment of the present application also provides a face image processing device for implementing the face image processing method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more face image processing device embodiments provided below can refer to the limitations on the face image processing method above, and will not be repeated here.
[0192] like Figure 7 As shown, the present application also provides a face image processing device. The device includes:
[0193] The data acquisition module 200 is used to acquire an original image and a makeup target image corresponding to the original image;
[0194] The first processing module 400 is used to input the original image into a preset trained first network model to obtain a color image;
[0195] The second processing module 600 is used to input the color image and the makeup target image into a preset trained second network model to obtain a color image after facial makeup transfer.
[0196] The above-mentioned facial image processing device obtains an original image and a makeup target image corresponding to the original image; inputs the original image into a preset trained first network model to obtain a color image; inputs the color image and the makeup target image into a preset trained second network model to obtain a color image after facial makeup migration. In the whole process, the first network model (aging model) and the second network model (makeup model) are used to obtain the color image and the color image after facial makeup migration, respectively, without relying on manual annotation, and efficient facial makeup migration can be achieved.
[0197] In one embodiment, the first network model includes a feature extraction layer, an encoder, a residual block layer, a decoding layer, and an evaluator;
[0198] The feature extraction layer adopts the network structure of VGG-16, and the output of the feature extraction layer is used as the input of the evaluator in the first network model; the residual block layer includes a convolutional layer, a regularization layer and an activation function connected in sequence; the encoder and decoding layers respectively include multiple different convolutional module layers.
[0199] In one embodiment, the first processing module 400 is also used for the feature extraction layer to extract the feature map of the original image, the encoding layer encodes the feature map and sends it to the residual block layer, the residual block layer maintains the size of the feature map unchanged, and transmits it to the decoding layer, and the decoding layer performs an up-convolution operation on the received feature map to obtain a color image.
[0200] In one embodiment, the first processing module 400 is also used to obtain an original data set; arbitrarily select a picture from the original data set to obtain a sample picture; input the sample picture into the initial first network model to obtain a sample color picture set, the sample color picture set includes a plurality of sample color pictures after facial aging; input each sample color picture in the sample color picture set into the evaluator of the initial first network model to obtain an aging result corresponding to the sample color picture, and obtain the loss of the evaluator and the loss of the generator in the initial first network model according to the aging result; input each sample color picture in the sample color picture set into the initial first network model to obtain a secondary sample color picture set, the secondary sample color picture set includes The invention discloses a method for preparing a plurality of sample color images after secondary facial aging; obtaining the real age labels corresponding to the sample images, and obtaining the perceptual loss function in the initial first network model according to the sample color images after secondary facial aging and the real age labels; inputting the sample color image set into the encoder in the initial first network model to obtain the encoding result, subjecting the encoding result to residual processing and decoding processing to obtain the facial aging result, and obtaining the texture loss decoded in the initial first network model according to the facial aging result; iteratively training the initial first network model according to the loss of the evaluator and the loss of the generator, the perceptual loss and the texture loss, until the training objective optimization function of the trained first network model converges, and obtaining the trained first network model.
[0201] In one embodiment, the second processing module 600 is further used to input the sample image and the sample color image set into the generator of the initial second network model to obtain the target image set and the generated data set, and obtain the perceptual loss and the local histogram loss according to the target image set and the generated data set; input each target image in the target image set into the first discriminator in the initial second network model to obtain the first recognition result corresponding to each color image, and obtain the loss of the first discriminator in the initial second network model according to the first recognition result corresponding to each color image; input each target image in the target image set into the second discriminator in the initial second network model to obtain each The second recognition result corresponding to the color picture is used to obtain the loss of the second discriminator in the initial second network model according to the second recognition result corresponding to each color picture; each target picture in the target picture set is input into the generator in the initial second network model to obtain a restored generated picture set and a generated makeup picture set, and a cycle consistency loss is obtained according to the generated picture set and the makeup picture set; the initial second network model is iteratively trained according to the perceptual loss, the local histogram loss, the loss of the first discriminator, the loss of the second discriminator, and the cycle consistency loss until the training objective optimization function in the second network model converges to obtain a trained second network model.
[0202] In one of the embodiments, the evaluator in the first network model includes multiple discriminators, and the evaluation value output by the evaluator is obtained by concatenating the output results of the multiple discriminators.
[0203] Each module in the above-mentioned face image processing device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each module above.
[0204] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 8 As shown. The computer device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a facial image processing method is implemented.
[0205] Those skilled in the art will understand that Figure 8 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0206] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:
[0207] Obtain an original image and a makeup target image corresponding to the original image;
[0208] Inputting the original image into a preset trained first network model to obtain a color image, wherein the first network model includes an aging model;
[0209] The color image and the makeup target image are input into a preset trained second network model to obtain a color image after facial makeup transfer, and the second network model includes a makeup model.
[0210] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:
[0211] Obtain an original image and a makeup target image corresponding to the original image;
[0212] Inputting the original image into a preset trained first network model to obtain a color image, wherein the first network model includes an aging model;
[0213] The color image and the makeup target image are input into a preset trained second network model to obtain a color image after facial makeup transfer, and the second network model includes a makeup model.
[0214] In one embodiment, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the following steps:
[0215] Obtain an original image and a makeup target image corresponding to the original image;
[0216] Inputting the original image into a preset trained first network model to obtain a color image, wherein the first network model includes an aging model;
[0217] The color image and the makeup target image are input into a preset trained second network model to obtain a color image after facial makeup transfer, and the second network model includes a makeup model.
[0218] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.
[0219] A person skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing related hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods.
[0220] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0221] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.
Claims
1. A facial image processing method, characterized in that: The method comprises: Obtaining an original image and a makeup target image corresponding to the original image; Inputting the original image into a preset trained first network model to obtain a color image, wherein the first network model includes an aging model; Inputting the color image and the makeup target image into a preset trained second network model to obtain a color image after facial makeup transfer, wherein the second network model includes a makeup model; The training process of the first network model includes: obtaining an original data set; arbitrarily selecting a picture from the original data set to obtain a sample picture; inputting the sample picture into the initial first network model to obtain a sample color picture set, wherein the sample color picture set includes a plurality of sample color pictures after facial aging; inputting each sample color picture in the sample color picture set into the evaluator of the initial first network model to obtain an aging result corresponding to the sample color picture, and obtaining the loss of the evaluator and the loss of the generator in the initial first network model according to the aging result; inputting each sample color picture in the sample color picture set into the initial first network model to obtain a secondary sample color picture set, wherein the secondary sample color picture set includes a plurality of A sample color picture after secondary facial aging; obtaining a real age label corresponding to the sample picture, and obtaining a perceptual loss function in an initial first network model based on the sample color picture after secondary facial aging and the real age label; inputting the sample color picture set into an encoder in the initial first network model to obtain an encoding result, subjecting the encoding result to residual processing and decoding processing to obtain a facial aging result, and obtaining a texture loss decoded in the initial first network model based on the facial aging result; iteratively training the initial first network model based on the loss of the evaluator and the loss of the generator, the perceptual loss, and the texture loss until the training objective optimization function of the trained first network model converges, thereby obtaining a trained first network model; The training process of the second network model includes: inputting the sample pictures and the sample color picture set into the generator of the initial second network model to obtain the target picture set and the generated data set, and obtaining the perceptual loss and the local histogram loss according to the target picture set and the generated data set; inputting each target picture in the target picture set into the first discriminator in the initial second network model to obtain the first recognition result corresponding to each color image, and obtaining the loss of the first discriminator in the initial second network model according to the first recognition result corresponding to each color image; inputting each target picture in the target picture set into the second discriminator in the initial second network model to obtain the first recognition result corresponding to each color image The second recognition result corresponding to each of the color pictures is obtained, and the loss of the second discriminator in the initial second network model is obtained according to the second recognition results corresponding to each of the color pictures; each target picture in the target picture set is input into the generator in the initial second network model to obtain a restored generated picture set and a generated makeup picture set, and a cycle consistency loss is obtained according to the generated picture set and the makeup picture set; according to the perceptual loss, the local histogram loss, the loss of the first discriminator, the loss of the second discriminator and the cycle consistency loss, the initial second network model is iteratively trained until the training objective optimization function in the second network model converges, thereby obtaining a trained second network model.
2. The method according to claim 1, characterized in that The first network model includes a feature extraction layer, an encoder, a residual block layer, a decoding layer and an evaluator; The feature extraction layer adopts the VGG-16 network structure, and the output of the feature extraction layer serves as the input of the evaluator in the first network model; the residual block layer includes a convolution layer, a regularization layer, and an activation function connected in sequence; the encoder and the decoding layer each include multiple different convolution module layers.
3. The method according to claim 2, characterized in that Inputting the original image into a preset trained first network model to obtain a color image includes: The feature extraction layer extracts the feature map of the original image, the encoding layer encodes the feature map and sends it to the residual block layer, the residual block layer maintains the size of the feature map unchanged, and transmits it to the decoding layer, and the decoding layer performs an upconvolution operation on the received feature map to obtain a color image.
4. The method according to claim 2, characterized in that The evaluator in the first network model includes multiple discriminators, and the evaluation value output by the evaluator is obtained by splicing the output results of the multiple discriminators.
5. A facial image processing device, characterized in that: The device comprises: A data acquisition module, configured to acquire an original image and a makeup target image corresponding to the original image; A first processing module is configured to input the original image into a preset trained first network model to obtain a color image, wherein the first network model includes an aging model; a second processing module, configured to input the color image and the makeup target image into a preset trained second network model to obtain a color image after facial makeup transfer, wherein the second network model includes a makeup model; The training process of the first network model includes: obtaining an original data set; arbitrarily selecting a picture from the original data set to obtain a sample picture; inputting the sample picture into the initial first network model to obtain a sample color picture set, wherein the sample color picture set includes a plurality of sample color pictures after facial aging; inputting each sample color picture in the sample color picture set into the evaluator of the initial first network model to obtain an aging result corresponding to the sample color picture, and obtaining the loss of the evaluator and the loss of the generator in the initial first network model according to the aging result; inputting each sample color picture in the sample color picture set into the initial first network model to obtain a secondary sample color picture set, wherein the secondary sample color picture set includes a plurality of A sample color picture after secondary facial aging; obtaining a real age label corresponding to the sample picture, and obtaining a perceptual loss function in an initial first network model based on the sample color picture after secondary facial aging and the real age label; inputting the sample color picture set into an encoder in the initial first network model to obtain an encoding result, subjecting the encoding result to residual processing and decoding processing to obtain a facial aging result, and obtaining a texture loss decoded in the initial first network model based on the facial aging result; iteratively training the initial first network model based on the loss of the evaluator and the loss of the generator, the perceptual loss, and the texture loss until the training objective optimization function of the trained first network model converges, thereby obtaining a trained first network model; The training process of the second network model includes: inputting the sample pictures and the sample color picture set into the generator of the initial second network model to obtain the target picture set and the generated data set, and obtaining the perceptual loss and the local histogram loss according to the target picture set and the generated data set; inputting each target picture in the target picture set into the first discriminator in the initial second network model to obtain the first recognition result corresponding to each color image, and obtaining the loss of the first discriminator in the initial second network model according to the first recognition result corresponding to each color image; inputting each target picture in the target picture set into the second discriminator in the initial second network model to obtain the first recognition result corresponding to each color image The second recognition result corresponding to each of the color pictures is obtained, and the loss of the second discriminator in the initial second network model is obtained according to the second recognition results corresponding to each of the color pictures; each target picture in the target picture set is input into the generator in the initial second network model to obtain a restored generated picture set and a generated makeup picture set, and a cycle consistency loss is obtained according to the generated picture set and the makeup picture set; according to the perceptual loss, the local histogram loss, the loss of the first discriminator, the loss of the second discriminator and the cycle consistency loss, the initial second network model is iteratively trained until the training objective optimization function in the second network model converges, thereby obtaining a trained second network model.
6. The device according to claim 5, characterized in that The first network model includes a feature extraction layer, an encoder, a residual block layer, a decoding layer and an evaluator; the feature extraction layer adopts the VGG-16 network structure, and the output of the feature extraction layer serves as the input of the evaluator in the first network model; the residual block layer includes a convolutional layer, a regularization layer and an activation function connected in sequence; the encoder and the decoding layer each include multiple different convolutional module layers.
7. The device according to claim 6, characterized in that The first processing module is also used for the feature extraction layer to extract the feature map of the original image. The encoding layer encodes the feature map and sends it to the residual block layer. The residual block layer maintains the size of the feature map unchanged and transmits it to the decoding layer. The decoding layer performs an up-convolution operation on the received feature map to obtain a color image.
8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 4 are implemented.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.
10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.
