Virtual beauty makeup generation method, system and device and storage medium

Abstract

The embodiment of the invention discloses a virtual beauty makeup generation method, system and device and a storage medium. According to the technical scheme provided by the embodiment of the invention, the method comprises the steps: obtaining a first face source image to be subjected to beauty makeup processing and a first beauty makeup image referred to by the beauty makeup processing of the first face source image; inputting the first beauty makeup image into a pre-constructed generator of a generative adversarial network model, and outputting a corresponding first beauty makeup material; and rendering the first beauty makeup material to a corresponding position of the first face source image, and generating a first beauty makeup processing result of the first face source image. By adopting the technical means, the flexibility of virtual beauty makeup can be improved, any beauty makeup material required by the user is extracted in real time and rendered to the face image of the user, personalized setting of the virtual beauty makeup is realized, personalized processing requirements of the user on different beauty makeup materials are met, and the use experience of the user is improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of image processing technology, and in particular to a virtual makeup generation method, system, device and storage medium. Background Art

[0002] At present, with the rise of computer vision and artificial intelligence technology, virtual makeup technology based on makeup materials has been widely used in many fields. For example, in a virtual makeup scene, users can capture their facial images through a camera, automatically identify the eyes, lips, cheeks and other areas based on the user's facial features, and then combine pre-extracted makeup materials (such as eyelashes, eye shadow, lipstick and blush, etc.) to perform accurate makeup trials. According to actual needs, the application end can extract the corresponding makeup materials from the relevant images in advance through manual or image recognition methods, so that users can match their makeup during virtual makeup.

[0003] However, when using makeup materials for virtual makeup, the makeup materials are usually pre-extracted by manual or image recognition, and the extraction process is cumbersome and complicated. Moreover, users can only use the makeup materials provided by the application for virtual makeup, and the selection of makeup materials is relatively fixed, which makes it difficult to meet the personalized needs of users for the use of makeup materials, resulting in a lack of flexibility in virtual makeup and poor user experience. Summary of the invention

[0004] The embodiments of the present application provide a virtual makeup generation method, system, device and storage medium, which can accurately collect makeup materials based on the makeup facial images uploaded by users, and integrate the collected makeup materials into the user's facial images for virtual makeup, thereby improving the flexibility of virtual makeup and solving the technical problem that the makeup materials of virtual makeup do not meet user needs.

[0005] In a first aspect, an embodiment of the present application provides a virtual makeup generation method, comprising:

[0006] Acquire a first source face image to be processed with beauty makeup and a first beauty makeup image as a reference for the beauty makeup processing of the first source face image;

[0007] Input the first beauty-makeup image into the generator of the pre-built generative adversarial network model, and output the corresponding first beauty-makeup material. The training samples of the generative adversarial network model include the second face source image, the second beauty-makeup image for reference of the second face source image for beauty-makeup processing, and the beauty-processed image corresponding to the second face source image. The second beauty-makeup image is used to train the generator of the generative adversarial network model, and output the corresponding second beauty-makeup material. The second beauty-makeup material is used to render to the second face source image to generate a second beauty-processing result. The beauty-processed image and the second beauty-processing result are used to train the discriminator of the generative adversarial network model, so as to iteratively adjust the generative adversarial network model based on the discrimination result of the discriminator.

[0008] The first beauty makeup material is rendered to a corresponding position of the first face source image to generate a first beauty makeup processing result of the first face source image.

[0009] In a second aspect, an embodiment of the present application provides a virtual makeup generation system, including:

[0010] An acquisition module configured to acquire a first face source image to be processed with beauty makeup and a first beauty makeup image as a reference for the first face source image to be processed with beauty makeup;

[0011] A material extraction module is configured to input the first beauty image into a generator of a pre-built generative adversarial network model, and output the corresponding first beauty material, wherein the training samples of the generative adversarial network model include a second face source image, a second beauty image for reference in beauty processing of the second face source image, and a beauty-processed image corresponding to the second face source image, wherein the second beauty image is used to train the generator of the generative adversarial network model, and output the corresponding second beauty material, wherein the second beauty material is used to render to the second face source image to generate a second beauty processing result, and the beauty-processed image and the second beauty processing result are used to train a discriminator of the generative adversarial network model, so as to iteratively adjust the generative adversarial network model based on the discrimination result of the discriminator;

[0012] The beauty makeup generation module is configured to render the first beauty makeup material to a corresponding position of the first face source image to generate a first beauty makeup processing result of the first face source image.

[0013] In a third aspect, an embodiment of the present application provides a virtual makeup generation device, including:

[0014] memory and one or more processors;

[0015] The memory is configured to store one or more programs;

[0016] When the one or more programs are executed by the one or more processors, the one or more processors implement the virtual makeup generation method as described in the first aspect.

[0017] In a fourth aspect, an embodiment of the present application provides a non-volatile computer-readable storage medium, which stores computer-executable instructions, and the computer-executable instructions, when executed by a computer processor, are configured to execute the virtual makeup generation method as described in the first aspect.

[0018] In a fifth aspect, an embodiment of the present application provides a computer program product, which includes instructions. When the instructions are executed on a computer or a processor, the computer or the processor executes the virtual makeup generation method as described in the first aspect.

[0019] The embodiment of the present application obtains a first face source image to be beauty-processed and a first beauty image for reference in the beauty-processing of the first face source image; inputs the first beauty image into a generator of a pre-constructed generative adversarial network model, and outputs a corresponding first beauty material. The training samples of the generative adversarial network model include a second face source image, a second beauty image for reference in the beauty-processing of the second face source image, and a beauty-processed image corresponding to the second face source image. The second beauty image is used to train the generator of the generative adversarial network model, and the corresponding second beauty material is output. The second beauty material is used to render to the second face source image to generate a second beauty processing result. The beauty-processed image and the second beauty processing result are used to train a discriminator of the generative adversarial network model, so as to iteratively adjust the generative adversarial network model based on the discrimination result of the discriminator; the first beauty material is rendered to the corresponding position of the first face source image, and a first beauty processing result of the first face source image is generated. By adopting the above-mentioned technical means, the first beauty image for reference in beauty processing of the first face source image is extracted through the pre-built generative adversarial network model, and the corresponding first beauty material is obtained. The first beauty material is then rendered to the corresponding position of the first face source image to generate the corresponding first beauty processing result, thereby improving the flexibility of virtual makeup, extracting and rendering any beauty material required by the user to the user's face image in real time, realizing personalized settings of virtual makeup, meeting the user's personalized processing needs for different beauty materials, and improving the user's usage experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 is a flow chart of a virtual makeup generation method provided in an embodiment of the present application;

[0021] Figure 2 It is a training flow chart of a generative adversarial network model in an embodiment of the present application;

[0022] Figure 3 is a processing flow chart of generating an adversarial network model in an embodiment of the present application;

[0023] Figure 4This is a flowchart of beauty makeup processing for a user's bare face in an embodiment of the present application;

[0024] Figure 5 is a schematic diagram of the beauty makeup effect of a user's bare face in an embodiment of the present application;

[0025] Figure 6 It is a structural schematic diagram of a virtual makeup generation system provided in an embodiment of the present application;

[0026] Figure 7 It is a structural schematic diagram of a virtual beauty makeup generation device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical scheme and advantages of the present application clearer, the specific embodiments of the present application are further described in detail below in conjunction with the accompanying drawings. It is understood that the specific embodiments described herein are only used to explain the present application, rather than to limit the present application. It should also be noted that, for the convenience of description, only the part related to the present application but not all the contents are shown in the accompanying drawings. Before discussing the exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow chart describes each operation (or step) as a sequential process, many of the operations therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of each operation can be rearranged. The process can be terminated when its operation is completed, but it can also have additional steps not included in the accompanying drawings. The process can correspond to a method, a function, a procedure, a subroutine, a subprogram, etc.

[0028] The virtual makeup generation method provided in the present application aims to extract a first makeup image for reference for makeup processing of a first face source image through a pre-built generative adversarial network model, obtain a corresponding first makeup material, and then render the first makeup material to a corresponding position of the first face source image, generate a corresponding first makeup processing result, thereby improving the flexibility of virtual makeup, extracting and rendering any makeup material required by the user to the user's face image in real time, thereby realizing personalized settings of virtual makeup.

[0029] With the rise of computer vision and artificial intelligence technologies, beauty material extraction technology has been widely used in many fields. For example, in the virtual makeup experience, users can capture their facial images through the camera and add beauty effects in real time on smart devices. This technology can automatically identify areas such as eyes, lips, and cheeks based on the user's facial features, and then accurately try out makeup with pre-extracted beauty materials (such as eyelashes, eye shadow, lipstick, and blush). In terms of beauty material creation, the application end can extract the most popular beauty materials from relevant beauty images through manual or image recognition for users to choose.

[0030] However, in related virtual makeup generation solutions, makeup materials are usually pre-extracted by manual or image recognition, and the extraction process is cumbersome and complicated. Moreover, users can only use the makeup materials provided by the application for virtual makeup, and the selection of makeup materials is relatively fixed, which makes it difficult to meet the personalized needs of users for the use of makeup materials, resulting in a lack of flexibility in virtual makeup and poor user experience.

[0031] In addition, facial features (such as eyes, eyebrows, mouth, nose, face shape, etc.) may change under different lighting, angles, expression changes, facial surface texture and other conditions, which makes it more difficult to extract facial features. The makeup effects generated subsequently may not appear realistic enough due to technical limitations. For example, the makeup may not blend perfectly with the user's facial contours, or the interaction between makeup and skin (such as the matching of foundation and skin color, and the fit of lipstick and lips) is not natural enough. On the other hand, in the extracted face image, the posture, angle and expression of the face may change, making the accuracy and stability of makeup extraction complicated.

[0032] Based on this, a virtual makeup generation method is provided in the embodiment of the present application to solve the technical problem that the makeup materials of virtual makeup do not meet the needs of users and improve the flexibility of virtual makeup. The makeup materials are accurately extracted through the pre-trained generative adversarial network model, and the makeup materials are rendered onto the user's face image to achieve a real, natural and stable virtual makeup effect.

[0033] Among them, Generative Adversarial Networks (GANs) is a deep learning model that generates new data similar to the training data through adversarial training of two neural networks. GANs consists of two main parts: the generator and the discriminator.

[0034] The goal of the generator is to generate samples that look real. It usually receives a random noise as input and then generates simulated samples. The generator usually uses a transposed convolutional neural network to generate images. The input is a low-dimensional random noise vector, which is gradually upsampled through a series of deconvolution layers to finally generate high-dimensional image data.

[0035] The goal of the discriminator is to distinguish between real samples and generated samples. It receives a sample (which may be a real sample or a generated sample) as input and outputs the probability that the sample is a real sample. The discriminator is usually implemented using a convolutional neural network, which has a structure similar to that of a traditional classification network. The input is an image, and after a series of convolutional layers and pooling layers, a probability value is output, indicating the probability that the input image is a real sample.

[0036] The working principle of GANs can be summarized as a "game" process. During the training process, the generator and the discriminator compete with each other: the generator continuously improves the quality of the generated samples in order to deceive the discriminator. The discriminator continuously improves its discrimination ability to better distinguish between real samples and generated samples. This process eventually tends to a balance, where the generator can generate high-quality samples and the discriminator's judgment ability is also optimal. The training goal of GANs is to alternately minimize the loss functions of the generator and the discriminator. By optimizing these loss functions, the generator and the discriminator continuously improve their performance in adversarial training.

[0037] Example:

[0038] Figure 1 A flowchart of a virtual makeup generation method provided in an embodiment of the present application is given. The virtual makeup generation method provided in this embodiment can be executed by a virtual makeup generation device, which can be implemented by software and / or hardware. The virtual makeup generation device can be composed of two or more physical entities, or it can be composed of one physical entity. Generally speaking, the virtual makeup generation device can be a terminal device such as a computer, a mobile phone, or a tablet.

[0039] The following description is made by taking the virtual makeup generation device as an example of the subject for executing the virtual makeup generation method. Figure 1 , the virtual makeup generation method specifically includes:

[0040] S110, obtaining a first face source image to be processed with beauty makeup and a first beauty makeup image as a reference for the beauty makeup processing of the first face source image.

[0041] In the virtual makeup generation process, the present application obtains a face image to be processed by beauty makeup, which is defined as the first face source image. In addition, a beauty makeup image for reference in beauty makeup processing is obtained, which is defined as the first beauty makeup image. Users can take selfies through an application on a smart device (such as a smart phone, tablet computer, or personal computer equipped with a camera) to obtain their own facial images. The collected image is preprocessed by preprocessing the user's facial image, such as resizing, cropping to a suitable ratio, and removing noise to ensure compatibility with the beauty makeup material, so as to obtain the first face source image. Optionally, the first face source image can also come from any face image uploaded by the user or a face video with a face image. In addition, the user can upload any beauty makeup image that needs to be referenced for the current virtual beauty makeup processing. These beauty makeup images may include beauty makeup materials such as eye shadow, lipstick, blush, etc. of different styles.

[0042] Based on the first face source image uploaded by the user and the second face source image, the validity of the image can be judged, such as whether there is a face, whether the image is clear, whether the format meets the standard, etc. After the validity judgment is completed, the beauty material extraction can be performed to realize virtual beauty processing and generate a virtual beauty effect that meets the user's expectations.

[0043] S120. Input the first beauty image into the generator of the pre-built generative adversarial network model, and output the corresponding first beauty material. The training samples of the generative adversarial network model include the second face source image, the second beauty image for reference of the second face source image for beauty processing, and the beauty-processed image corresponding to the second face source image. The second beauty image is used to train the generator of the generative adversarial network model, and output the corresponding second beauty material. The second beauty material is used to render to the second face source image to generate a second beauty processing result. The beauty-processed image and the second beauty processing result are used to train the discriminator of the generative adversarial network model, so as to iteratively adjust the generative adversarial network model based on the discrimination result of the discriminator.

[0044] Furthermore, based on the first face source image and the first beauty makeup image obtained above, a generative adversarial network (GANs) model is used to extract beauty makeup materials from the first beauty makeup image for subsequent virtual beauty makeup rendering in combination with the first face source image.

[0045] By providing a pre-built GANs model, the GANs model can identify and extract beauty materials that match the beauty reference image (i.e., the first beauty image). The training of the model involves a large number of second face source images, second beauty images, and corresponding beauty processed images, wherein the beauty processed image is a face image captured after the face corresponding to the second face source image is subjected to real beauty makeup, that is, a face image with a real beauty effect. In addition, the beauty processed image can also be a face image obtained by performing corresponding image processing (such as using a P-image software to perform P-image) operations on the corresponding second face source image based on the beauty material effect of the second beauty image referenced by the second face source image.

[0046] Based on the above pre-trained input of the first beauty image to the generator GANs model, when the uploaded first beauty image (i.e. the beauty material style that the user wishes to apply) is received, the key points of the face of the first beauty image are first extracted, such as the key points of the upper and lower eyelids in the eye area, the key points of the nose tip, nose bridge and nose wings in the nose area, and the corners of the mouth, the inner and outer arc curve contour points of the upper and lower lips in the mouth area, etc. Then, the mask area of ​​the facial features is extracted using the image segmentation model, and the positions of the facial key points are used to cut out the facial features such as eyes, mouth and eyebrows from the first beauty image, and the resolution is consistent with the resolution of the corresponding beauty material.

[0047] The segmented facial features image is then input into the generator of the GANs model. The generator generates corresponding beauty materials based on the input facial features image, which are defined as the first beauty materials. A beauty material may include different beauty elements such as eye shadow, lipstick, blush, etc. The generator generates beauty materials that match the input beauty image through a series of complex calculations (such as convolution, deconvolution, upsampling, etc.). These materials are close to the first beauty image in shape, color and texture.

[0048] Specifically, refer to Figure 2 , the training process of the generative adversarial network model includes:

[0049] S1001, inputting the second beauty makeup image into the generator of the generative adversarial network model, and outputting the corresponding second beauty makeup material;

[0050] S1002, rendering the second beauty makeup material to a corresponding position of the second face source image to generate a second beauty makeup processing result;

[0051] S1003, inputting the second beauty-processing result and the beauty-processing image into a discriminator of a generative adversarial network model, and outputting a discrimination result;

[0052] S1004. Calculate the loss function of the generative adversarial network model based on the discrimination result, and iteratively adjust the model parameters of the generator and the discriminator of the generative adversarial network model based on the loss function until the generative adversarial network model converges.

[0053] In the training process of the GANs model, a large number of second face source images, second beauty images, and corresponding beauty processed images are used as training samples. These samples are used to train the generator and the discriminator, wherein the second beauty image is used to train the generator so that it can generate beauty materials that match the second beauty image. The second beauty material generated by the generator is rendered onto the second face source image to generate a second beauty processing result, which is used to compare with the real beauty processing image. The beauty processing image and the second beauty processing result are used together to train the discriminator, whose task is to distinguish between the real beauty processing image and the beauty processing result generated by the generator.

[0054] Exemplarily, in the data preparation stage of model training, the following data is prepared:

[0055] (1) Reference Image: This image data refers to the second beauty image, which is used as the input image for GAN to extract beauty materials.

[0056] (2) Source Image: refers to the second source face image. The second source face image can be a plain face image, that is, a face image without any makeup effect. It can also be any face image that needs to be processed with makeup.

[0057] (3) Target Image: refers to a real beauty-processed image, which can be obtained by extracting beauty materials from Reference Image and pasting them to the corresponding position of Source Image. In other words, the face in Source Image and the face in Target Image come from the same person. The difference is that the former is without makeup, while the latter has the real makeup effect of Reference Image. In addition, the image can also be obtained by manually processing the second face source image with reference to the beauty material effect of Reference Image, or by manually applying makeup to the face of the second face source image and collecting the image to obtain the beauty-processed image. There are many ways to obtain beauty-processed images, and there is no fixed restriction here.

[0058] Optionally, the second makeup image Reference Image and the second face source image Souce Image can be downloaded from an open source dataset or search engine on the Internet. As long as a face image is without makeup, it meets the Source Image requirement. On the contrary, if its face is with makeup effect, it meets the Source Image requirement. As for the production of the makeup processing result Target Image, the makeup designer can refer to the Reference Image to produce the makeup material, and then render the produced makeup material between different bare face A and B to obtain the corresponding makeup face A. ′ and B ′ , which is the beauty-processed image Target Image. So in this dataset, the plain faces A and B are the second face source images Source Image, and the beauty-processed face A ′ and B′ is the beauty processed image Target Image.

[0059] By obtaining a large amount of second makeup images Reference Images and second face source images Souce Images on the Internet, it is helpful for the GANs model to train the subtle eyelash texture and lipstick differences on the face. At the same time, the makeup processing images Target Images are produced by makeup designers to ensure the authenticity, naturalness and reliability of the training results.

[0060] Among them, refer to Figure 3 For the training samples prepared above, the generative adversarial network model first extracts the positions of the key points of the face in the second beauty makeup image Reference Image, such as the key points of the upper and lower eyelids in the eye area, the key points of the nose tip, nose bridge and nose wings in the nose area, and the corners of the mouth, the inner and outer arcs of the upper and lower lips in the mouth area. Then, the image segmentation model is used to extract the mask area of ​​the facial features, and the positions of the key points of the face are used to cut out the facial features such as eyes, mouth and eyebrows from the first beauty makeup image, and the resolution is consistent with the resolution of the corresponding beauty makeup material.

[0061] Then, based on the above segmented facial features images, the generator and discriminator are optimized with each other through adversarial training. The generator attempts to generate realistic makeup images, while the discriminator strives to distinguish between the generated images and real makeup images.

[0062] Among them, the task of the generator is to receive an image of facial features with makeup (that is, the facial features image cut out according to the key points of the face as mentioned above) as input, and output a corresponding beauty makeup material, that is, the second beauty makeup material.

[0063] Optionally, inputting the second beauty makeup image into a generator of a generative adversarial network model and outputting a corresponding second beauty makeup material includes:

[0064] Inputting the second beauty-makeup image into a generator of a generative adversarial network model, and extracting facial feature segmentation regions from the second beauty-makeup image based on the generator;

[0065] The U-shaped network architecture based on the generator processes the facial feature segmentation areas extracted from the second beauty makeup image to generate the corresponding second beauty makeup material.

[0066] Among them, the second beauty material is in .png format with 4 channels, the first 3 channels are RGB channels, and the last channel is a mask channel. The generator adopts the U-Ne architecture. The U-Net architecture is "U" shaped and consists of three parts: encoder (downsampling path), decoder (upsampling path) and jump connection. This design enables U-Net to effectively retain low-level spatial information and perform efficient segmentation when processing image segmentation tasks. The encoder is responsible for extracting the features of the input image. It consists of a series of convolutional layers and pooling layers. Multiple convolutional layers (usually 3×3 convolutions) are used, each followed by a ReLU activation function. The convolutional layer is used to extract features from the image. The pooling layer reduces the size of the feature map through the maximum pooling layer while increasing the abstraction of the features. After each layer of pooling, the depth of the feature map increases, and more complex features are gradually extracted. The decoder is responsible for restoring the features extracted by the encoder to the spatial resolution of the original image and generating a predicted label for each pixel. The decoder uses a deconvolution (Transposed Convolution) layer or an upsampling layer to gradually restore the spatial resolution of the feature map. Convolutional layers are used to further extract and integrate features in order to generate the final segmentation map. Skip connections are a key part of the U-Net architecture. It concatenates the output of each convolutional block in the encoder with the output of the corresponding position in the decoder. Doing so can effectively preserve detail information, help restore spatial resolution, and reduce blur that may occur during the segmentation process. Skip connections enable U-Net to restore fine-grained spatial information, which is particularly suitable for fine-grained segmentation tasks. It uses skip connections to preserve the details of the input image during the generation process, making the generated makeup effect more realistic.

[0067] like Figure 3 As shown, the facial features image in the second beauty material Reference_image cut out according to the key points of the face is used as input and respectively input into the generator network of the corresponding U-Net architecture to output the predicted beauty material image, such as eyelashes, lipstick, etc.

[0068] Then, using the Pytorch3D differentiable renderer, based on the face mesh area divided by the recognized face key points, the second beauty material is pasted to the corresponding second face source image Source Image, and the beauty face processed by the model is obtained, which is defined as the second beauty processing result. The mesh generated based on the face key point detection can determine the shape of the face and the position of the facial features. Then, a Meshes object is created in the Pytorch3D differentiable renderer, which contains the vertices, faces and textures of the face mesh area Face Mesh. By configuring the renderer parameters, such as camera position, lighting conditions, etc. In addition, shadows, highlights and other effects can be added to enhance the realism of the rendering. Then, using the Pytorch3D differentiable renderer, the face mesh area with the second beauty material texture is rendered to the corresponding position of the second face source image, thereby generating a beauty face image after the GANs model beauty processing, that is, the second beauty processing result.

[0069] The above-mentioned second beauty processing result and the corresponding beauty processing image are further input into the discriminator of the generative adversarial network model. The task of the discriminator is to determine whether the input image is a real facial image with makeup, that is, to determine whether the second beauty processing result and the corresponding beauty processing image are the same. The discriminator and the generator are in an adversarial state, and their discrimination accuracy is continuously optimized. The discriminator uses a convolutional neural network (CNN) for image classification and outputs a value to indicate whether the input second beauty processing result is a beauty processing image, that is, the discrimination result. Specifically, the discriminator will classify the second beauty processing result and the corresponding beauty processing image it receives, and output a probability value indicating the probability that the second beauty processing result is a real image (that is, a beauty processing image). This probability value is the discrimination result. The loss function of the generative adversarial network model can be calculated through the discrimination result of the discriminator, so as to iteratively adjust the model parameters of the generator and the discriminator of the generative adversarial network model based on the loss function until the generative adversarial network model converges.

[0070] Optionally, the loss function includes an adversarial loss calculated based on the beauty-processed image and the second beauty-processing result. The adversarial loss (GAN Loss) is trained by inputting the real beauty-processed image Target Image and the second beauty-processing result generated by the model into the discriminator respectively. The goal of the generator is to minimize the difference between the generated image and the real image, that is, to generate a makeup effect that is as realistic as possible. The goal of the discriminator is to maximize the ability to distinguish between the generated image and the real image, that is, to make the generated image as difficult to identify as fake as possible.

[0071] Optionally, the loss function also includes at least one of perceptual loss, histogram distribution loss and pixel level loss, the perceptual loss is calculated based on the second face source image and the second beauty processing result, the histogram distribution loss is calculated based on the pixel values ​​of the facial feature segmentation area of ​​the beauty-processed image and the second beauty processing result, and the pixel level loss is calculated based on the facial feature segmentation area of ​​the second face source image and the second beauty processing result.

[0072] Since the second makeup result and the second face source image are from the same person, the face ID needs to be consistent. Therefore, the perceptual loss of the VGG-16 model pre-trained on ImageNet is used to measure the difference between them and the corresponding perceptual loss is calculated.

[0073] In addition, according to the mask of the facial features segmentation, the RGB pixel values ​​of the facial features in the second beauty image Reference Image and the generated second beauty processing result are extracted respectively, and their histogram distribution loss Hisogramloss is calculated.

[0074] If the second face source image Source Image has a corresponding beauty-processed image Target Image, then the pixel-level loss value of the second face source image SourceImage and the beauty-processed image Target Image is calculated accordingly based on the facial mask of the second face source image Source Image as the index.

[0075] Furthermore, when the loss function includes multiple types of losses, iteratively adjusting model parameters of the generator and the discriminator of the generative adversarial network model based on the loss function includes:

[0076] The model parameters of the generator and discriminator of the generative adversarial network model are calculated based on the sum of the losses of each type, and the calculated model parameters are updated to the generator and discriminator of the generative adversarial network model accordingly.

[0077] By adding all of the above and updating the model parameters of the generator and discriminator through back-propagation calculation until the set convergence conditions are met, the training of the generative adversarial network model is completed.

[0078] It is understandable that if the discriminator can accurately identify the difference between the generated beauty processing results and the real beauty processing images, then the generator needs to further adjust its parameters to generate more realistic beauty materials. At the same time, the discriminator also needs to adjust its parameters according to its classification accuracy to improve its identification ability. The model training is completed by iteratively adjusting the model parameters until the generator can generate sufficiently realistic beauty materials and the discriminator can no longer effectively distinguish between the real beauty processing images and the generated beauty processing results.

[0079] Through a large number of training samples and iterative training, the GANs model can learn the shape, color, texture and other characteristics of beauty elements, and generate beauty materials that match the original beauty images. This not only improves the flexibility and personalization of virtual beauty, but also provides users with more natural and realistic beauty effects.

[0080] Based on the generator of the above-mentioned pre-trained generative adversarial network model, referring to the method of extracting beauty materials in the model training process, the first beauty material can be extracted corresponding to the first beauty image of the input model for subsequent virtual beauty processing.

[0081] S130: Rendering a first beauty makeup material to a corresponding position of the first face source image to generate a first beauty makeup processing result of the first face source image.

[0082] Afterwards, referring to the above-mentioned method of generating the second beauty processing result, based on the use of the Pytorch3D differentiable renderer, the first beauty material can be accurately rendered to the corresponding position of the first face source image (such as eyes, lips, cheeks, etc.) to generate the first beauty processing result of the first face source image.

[0083] The step of rendering the first makeup material to a corresponding position of the first face source image includes:

[0084] The positions of key facial points corresponding to the first beauty makeup material are determined, and based on the positions of the key facial points, the first beauty makeup material is overlaid to corresponding positions of the first face source image.

[0085] Using the Pytorch3D differentiable renderer, based on the face grid area divided by the facial key points recognized in the first face source image, the first beauty material is pasted into the corresponding first face source image, so as to obtain the first beauty processing result.

[0086] Through precise rendering technology, the extracted beauty materials can be naturally integrated into the user's facial image to generate virtual beauty effects that meet the user's personalized needs. At the same time, the flexibility and user experience of virtual beauty are improved, allowing users to freely choose and try different makeup styles according to their preferences.

[0087] For example, refer to Figure 4 During the live broadcast, users can upload their favorite facial makeup (i.e., beauty images) to the background of the live broadcast application. After receiving the image, the background beauty material extraction module based on the GANs model extracts the features of the eyes, pupils, eyebrows, lips and other areas, and then inputs them into the corresponding GANs model for inference and calculation, and outputs specific eyelashes, pupils, eyebrows and lipstick beauty materials. This process only extracts beauty materials once for each uploaded image. When the user is broadcasting live, the program will obtain the user's bare face video stream through the camera and decode it into a picture sequence, and then render it in the rendering module together with the aforementioned beauty materials to obtain a makeup face picture sequence and encode it into a makeup face video stream.

[0088] like Figure 5 As shown in the figure, after extracting the makeup materials of the makeup image through the GANs model, it can be applied to the user's bare face to obtain the corresponding makeup face. The GANs model dynamically generates makeup suitable for each user by learning from a large amount of facial makeup data, which can avoid the problem of inconsistent performance on different skin colors, skin qualities, and facial shapes. Secondly, the GANs model can generate high-quality, realistic images or data through adversarial training of the generator and the discriminator. For the extraction of makeup materials, it can generate real and natural makeup effects (such as lipstick, eye shadow, blush, etc.), making the virtual makeup trial more realistic and close to the actual makeup effect of the user. This allows virtual makeup processing to generate delicate texture details, such as skin texture and the degree of integration of makeup with skin, providing natural makeup effects.

[0089] In the above, a first face source image to be beautified and a first beauty image for reference in the beauty processing of the first face source image are obtained; the first beauty image is input into a generator of a pre-constructed generative adversarial network model, and a corresponding first beauty material is output. The training samples of the generative adversarial network model include a second face source image, a second beauty image for reference in the beauty processing of the second face source image, and a beauty-processed image corresponding to the second face source image. The second beauty image is used to train the generator of the generative adversarial network model, and the corresponding second beauty material is output. The second beauty material is used to render to the second face source image to generate a second beauty processing result. The beauty-processed image and the second beauty processing result are used to train the discriminator of the generative adversarial network model, so as to iteratively adjust the generative adversarial network model based on the discrimination result of the discriminator; the first beauty material is rendered to the corresponding position of the first face source image, and a first beauty processing result of the first face source image is generated. By adopting the above-mentioned technical means, the first beauty image for reference in beauty processing of the first face source image is extracted through the pre-built generative adversarial network model, and the corresponding first beauty material is obtained. The first beauty material is then rendered to the corresponding position of the first face source image to generate the corresponding first beauty processing result, thereby improving the flexibility of virtual makeup, extracting and rendering any beauty material required by the user to the user's face image in real time, realizing personalized settings of virtual makeup, meeting the user's personalized processing needs for different beauty materials, and improving the user's usage experience.

[0090] Based on the above embodiments, Figure 6 This is a schematic diagram of the structure of a virtual makeup generation system provided by this application. Figure 6 The virtual makeup generation system provided in this embodiment specifically includes: an acquisition module 21, a material extraction module 22 and a makeup generation module 23.

[0091] The acquisition module 21 is configured to acquire a first face source image to be processed with beauty makeup and a first beauty makeup image as a reference for the first face source image to be processed with beauty makeup;

[0092] The material extraction module 22 is configured to input the first beauty makeup image into the generator of the pre-built generative adversarial network model, and output the corresponding first beauty makeup material. The training samples of the generative adversarial network model include the second face source image, the second beauty makeup image for reference in the beauty makeup processing of the second face source image, and the beauty makeup processed image corresponding to the second face source image. The second beauty makeup image is used to train the generator of the generative adversarial network model, and output the corresponding second beauty makeup material. The second beauty makeup material is used to render to the second face source image to generate a second beauty makeup processing result. The beauty makeup processed image and the second beauty makeup processing result are used to train the discriminator of the generative adversarial network model, so as to iteratively adjust the generative adversarial network model based on the discrimination result of the discriminator.

[0093] The beauty makeup generation module 23 is configured to render the first beauty makeup material to a corresponding position of the first face source image, and generate a first beauty makeup processing result of the first face source image.

[0094] Specifically, the training process of the generative adversarial network model includes:

[0095] Input the second beauty makeup image into the generator of the generative adversarial network model, and output the corresponding second beauty makeup material;

[0096] Rendering the second beauty makeup material to a corresponding position of the second face source image to generate a second beauty makeup processing result;

[0097] Inputting the second beauty processing result and the beauty processing image into the discriminator of the generative adversarial network model, and outputting the discrimination result;

[0098] The loss function of the generative adversarial network model is calculated based on the discrimination result, and the model parameters of the generator and discriminator of the generative adversarial network model are iteratively adjusted based on the loss function until the generative adversarial network model converges.

[0099] The loss function includes an adversarial loss calculated based on the beauty-processed image and the second beauty-processing result.

[0100] The loss function also includes at least one of perceptual loss, histogram distribution loss and pixel level loss. The perceptual loss is calculated based on the second face source image and the second beauty processing result. The histogram distribution loss is calculated based on the pixel values ​​of the facial feature segmentation area of ​​the beauty-processed image and the second beauty processing result. The pixel level loss is calculated based on the facial feature segmentation area of ​​the second face source image and the second beauty processing result.

[0101] When the loss function includes multiple types of losses, the model parameters of the generator and the discriminator of the generative adversarial network model are iteratively adjusted based on the loss function, including:

[0102] The model parameters of the generator and discriminator of the generative adversarial network model are calculated based on the sum of the losses of each type, and the calculated model parameters are updated to the generator and discriminator of the generative adversarial network model accordingly.

[0103] The second beauty makeup image is input into the generator of the generative adversarial network model, and the corresponding second beauty makeup material is output, including:

[0104] Inputting the second beauty-makeup image into a generator of a generative adversarial network model, and extracting facial feature segmentation regions from the second beauty-makeup image based on the generator;

[0105] The U-shaped network architecture based on the generator processes the facial feature segmentation areas extracted from the second beauty makeup image to generate the corresponding second beauty makeup material.

[0106] Specifically, the second beauty makeup image is input into the generator of the generative adversarial network model, and the corresponding second beauty makeup material is output, including:

[0107] Inputting the second beauty-makeup image into a generator of a generative adversarial network model, and extracting facial feature segmentation regions from the second beauty-makeup image based on the generator;

[0108] The U-shaped network architecture based on the generator processes the facial feature segmentation areas extracted from the second beauty makeup image to generate the corresponding second beauty makeup material.

[0109] In the above, a first face source image to be beautified and a first beauty image for reference in the beauty processing of the first face source image are obtained; the first beauty image is input into a generator of a pre-constructed generative adversarial network model, and a corresponding first beauty material is output. The training samples of the generative adversarial network model include a second face source image, a second beauty image for reference in the beauty processing of the second face source image, and a beauty-processed image corresponding to the second face source image. The second beauty image is used to train the generator of the generative adversarial network model, and the corresponding second beauty material is output. The second beauty material is used to render to the second face source image to generate a second beauty processing result. The beauty-processed image and the second beauty processing result are used to train the discriminator of the generative adversarial network model, so as to iteratively adjust the generative adversarial network model based on the discrimination result of the discriminator; the first beauty material is rendered to the corresponding position of the first face source image, and a first beauty processing result of the first face source image is generated. By adopting the above-mentioned technical means, the first beauty image for reference in beauty processing of the first face source image is extracted through the pre-built generative adversarial network model, and the corresponding first beauty material is obtained. The first beauty material is then rendered to the corresponding position of the first face source image to generate the corresponding first beauty processing result, thereby improving the flexibility of virtual makeup, extracting and rendering any beauty material required by the user to the user's face image in real time, realizing personalized settings of virtual makeup, meeting the user's personalized processing needs for different beauty materials, and improving the user's usage experience.

[0110] The virtual makeup generation system provided in the embodiment of the present application can be configured to execute the virtual makeup generation method provided in the above embodiment, and has corresponding functions and beneficial effects.

[0111] Based on the above practical example, the present application embodiment also provides a virtual makeup generation device, referring to Figure 7, the virtual beauty generation device includes: a processor 31, a memory 32, a communication module 33, an input device 34 and an output device 35. The memory, as a computer-readable storage medium, can be configured to store software programs, computer executable programs and modules, such as program instructions / modules corresponding to the virtual beauty generation method described in any embodiment of the present application (for example, the acquisition module, material extraction module and beauty generation module in the virtual beauty generation system). The communication module is configured to transmit data. The processor executes various functional applications and data processing of the device by running the software programs, instructions and modules stored in the memory, that is, realizing the above-mentioned virtual beauty generation method. The input device can be configured to receive input digital or character information, and generate key signal input related to the user settings and function control of the device. The output device may include a display device such as a display screen. The virtual beauty generation device provided above can be configured to execute the virtual beauty generation method provided in the above-mentioned embodiment, and has corresponding functions and beneficial effects.

[0112] On the basis of the above embodiments, the embodiments of the present application further provide a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores computer-executable instructions, wherein the computer-executable instructions are configured to execute a virtual makeup generation method when executed by a computer processor, and the storage medium may be any of various types of memory devices or storage devices. Of course, the non-volatile computer-readable storage medium provided in the embodiments of the present application, whose computer-executable instructions are not limited to the virtual makeup generation method described above, may also execute the related operations in the virtual makeup generation method provided in any embodiment of the present application.

[0113] On the basis of the above embodiments, the embodiments of the present application also provide a computer program product. The essence of the technical solution of the present application or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer program product is stored in a storage medium, and includes a number of instructions for enabling a computer device, a mobile terminal or a processor therein to execute all or part of the steps of the virtual beauty generation method described in each embodiment of the present application.

Claims

1. A virtual makeup generation method, characterized in that: include: Acquire a first source face image to be processed with beauty makeup and a first beauty makeup image as a reference for the beauty makeup processing of the first source face image; Input the first beauty-makeup image into a generator of a pre-built generative adversarial network model, and output a corresponding first beauty-makeup material. The training samples of the generative adversarial network model include a second face source image, a second beauty-makeup image for reference of the second face source image for beauty-makeup processing, and a beauty-processed image corresponding to the second face source image. The second beauty-makeup image is used to train the generator of the generative adversarial network model, and output a corresponding second beauty-makeup material. The second beauty-makeup material is used to render to the second face source image to generate a second beauty-processing result. The beauty-processed image and the second beauty-processing result are used to train a discriminator of the generative adversarial network model, so as to iteratively adjust the generative adversarial network model based on the discrimination result of the discriminator. The first beauty makeup material is rendered to a corresponding position of the first face source image to generate a first beauty makeup processing result of the first face source image.

2. The virtual makeup generation method according to claim 1, characterized in that: The training process of the generative adversarial network model includes: Inputting the second beauty makeup image into the generator of the generative adversarial network model, and outputting a corresponding second beauty makeup material; Rendering the second beauty makeup material to a corresponding position of the second face source image to generate the second beauty makeup processing result; Inputting the second beauty-processing result and the beauty-processing image into the discriminator of the generative adversarial network model, and outputting a discriminant result; A loss function of the generative adversarial network model is calculated based on the discrimination result, and model parameters of a generator and a discriminator of the generative adversarial network model are iteratively adjusted based on the loss function until the generative adversarial network model converges.

3. The virtual makeup generation method according to claim 2, characterized in that: The loss function includes an adversarial loss calculated based on the beauty-processed image and the second beauty-processing result.

4. The virtual makeup generation method according to claim 3, characterized in that: The loss function also includes at least one of perceptual loss, histogram distribution loss and pixel-level loss. The perceptual loss is calculated based on the second face source image and the second beauty processing result. The histogram distribution loss is calculated based on the pixel values ​​of the facial feature segmentation area of ​​the beauty-processed image and the second beauty processing result. The pixel-level loss is calculated based on the facial feature segmentation area of ​​the second face source image and the second beauty processing result.

5. The virtual makeup generation method according to claim 2, characterized in that: In a case where the loss function includes multiple types of losses, the iterative adjustment of model parameters of the generator and the discriminator of the generative adversarial network model based on the loss function includes: The model parameters of the generator and the discriminator of the generative adversarial network model are calculated based on the sum of the losses of each type by back propagation, and the calculated model parameters are updated to the generator and the discriminator of the generative adversarial network model accordingly.

6. The virtual makeup generation method according to claim 2, characterized in that: The step of inputting the second beauty makeup image into the generator of the generative adversarial network model and outputting the corresponding second beauty makeup material includes: Inputting the second beauty-makeup image into the generator of the generative adversarial network model, and extracting facial feature segmentation regions from the second beauty-makeup image based on the generator; The facial feature segmentation areas extracted from the second beauty makeup image are processed based on the U-shaped network architecture of the generator to generate corresponding second beauty makeup materials.

7. The virtual makeup generation method according to any one of claims 1 to 6, characterized in that: The rendering of the first makeup material to a corresponding position of the first face source image includes: Determine the positions of key facial points corresponding to the first beauty makeup material, and cover the first beauty makeup material to corresponding positions of the first face source image based on the positions of the key facial points.

8. A virtual makeup generation system, characterized in that: include: An acquisition module configured to acquire a first source face image to be processed with beauty makeup and a first beauty makeup image as a reference for the beauty makeup processing of the first source face image; A material extraction module is configured to input the first beauty-makeup image into a generator of a pre-built generative adversarial network model, and output a corresponding first beauty-makeup material, wherein the training sample of the generative adversarial network model includes a second face source image, a second beauty-makeup image for reference of the second face source image for beauty-makeup processing, and a beauty-processed image corresponding to the second face source image, wherein the second beauty-makeup image is used to train the generator of the generative adversarial network model, and output a corresponding second beauty-makeup material, wherein the second beauty-makeup material is used to render to the second face source image to generate a second beauty-processing result, and the beauty-processed image and the second beauty-processing result are used to train a discriminator of the generative adversarial network model, so as to iteratively adjust the generative adversarial network model based on the discrimination result of the discriminator; The beauty makeup generation module is configured to render the first beauty makeup material to a corresponding position of the first face source image to generate a first beauty makeup processing result of the first face source image.

9. A virtual makeup generation device, characterized in that: include: memory and one or more processors; The memory is configured to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the virtual makeup generation method as described in any one of claims 1-7.

10. A non-volatile computer-readable storage medium, characterized in that: The non-volatile computer-readable storage medium stores computer-executable instructions, which, when executed by a computer processor, are configured to execute the virtual makeup generation method according to any one of claims 1 to 7.

11. A computer program product, characterized in that The computer program product includes instructions, and when the instructions are executed on a computer or a processor, the computer or the processor executes the virtual makeup generation method as described in any one of claims 1 to 7.

Images