A Multi-Scale Digital Makeup Transfer Method and System

Abstract

The present invention belongs to the technical field of image style transfer, and specifically relates to a multi-scale digital makeup transfer method and system, including: obtaining different facial makeup images; using a multi-scale digital network model to perform makeup transfer on the facial makeup images to obtain a facial makeup image after makeup transfer; specifically, during the process of performing makeup transfer on the facial makeup images, extracting the content features, makeup styles, and background regions of the obtained different facial makeup images; performing makeup transfer according to the extracted content features of the facial makeup images to obtain a makeup transfer image, and the transferred makeup style is different from the makeup style of the facial makeup image to be subjected to makeup transfer; extracting the facial region of the obtained makeup transfer image; fusing the extracted background region and the facial region to obtain a facial makeup image after makeup transfer, thereby realizing the makeup transfer of the facial makeup image.

Description

Technical Field

[0001] The present invention belongs to the technical field of image style transfer, and particularly relates to a multi-scale digital makeup transfer method and system. Background Art

[0002] The statements in this part merely provide background technical information related to the present invention and do not necessarily constitute prior art.

[0003] Makeup is a common way of facial beauty for people, which involves multiple steps and techniques such as foundation concealer, eyeshadow, lipstick, and painting of complex patterns; converting the physical makeup application and removal process into digital makeup transfer to achieve one-key synthesis of facial makeup on a human face image has become a research hotspot in the field of computer vision image style transfer, and it has a wide range of applications in multiple fields such as photo beauty, video beautification, digital entertainment, live broadcast, virtual makeup try-on, etc.

[0004] Digital makeup transfer mainly adopts facial makeup transfer technology, including digital makeup application and digital makeup removal; transferring the makeup style from a made-up (or makeup-free) human face image to a makeup-free (or made-up) human face image, and processing the makeup of different parts of the face in a coherent and natural manner. Digital makeup application has two goals: transferring the low-frequency color features and high-frequency detail features of the facial makeup from the made-up image to the makeup-free image; and, maintaining the unchanged identity features of the human face in the makeup-free image during the makeup application process. Digital makeup removal refers to removing the makeup effect on a made-up human face image and restoring the made-up human face image to a makeup-free human face image. Digital makeup removal has two goals: accurately identifying and eliminating the makeup effect on the human face image, including color, texture, and pattern, and avoiding unnatural boundaries or transitions, color distortion, etc.; and, the human face after makeup removal should maintain its original identity features.

[0005] As understood by the inventor, the existing makeup transfer research can be divided into two categories: the first category is traditional makeup transfer technology based on image processing methods such as image gradient editing and physical operations, and the second category is makeup transfer technology based on deep learning methods such as deep neural networks. Traditional makeup transfer technology has the disadvantages of complex modeling and large computational amount during the makeup transfer process; the makeup transfer technology based on deep learning is the current mainstream makeup transfer method and has made certain progress; however, the makeup transfer effect for complex patterns generated by facial painting is not good, there is an obvious boundary between the transferred complex pattern and the human face image, and the makeup is distorted; there will also be problems such as color residue or background information change, and it is impossible to maintain the naturalness and background consistency of the images before and after makeup transfer. Summary of the Invention

[0006] To solve the above problems, the present invention proposes a multi-scale digital makeup transfer method and system, which uses a generative adversarial network model with an introduced multi-discriminator architecture to perform multi-scale digital makeup transfer, separately processes the face region, reduces the impact on the makeup transfer effect, ensures the consistency of the image background before and after makeup transfer, and generates a natural and realistic image after makeup transfer.

[0007] According to some embodiments, the first solution of the present invention provides a multi-scale digital makeup transfer method, which adopts the following technical solutions:

[0008] A multi-scale digital makeup transfer method includes:

[0009] Obtain different facial makeup images;

[0010] Use a multi-scale digital network model to perform makeup transfer on the facial makeup images to obtain the facial makeup images after makeup transfer; specifically, during the process of performing makeup transfer on the facial makeup images, extract the content features, makeup styles, and background regions of the obtained different facial makeup images; perform makeup transfer according to the extracted content features of the facial makeup images to obtain a makeup transfer image, and the transferred makeup style is different from the makeup style of the facial makeup image to be transferred; extract the face region of the obtained makeup transfer image; fuse the extracted background region and face region to obtain the facial makeup image after makeup transfer, and realize the makeup transfer of the facial makeup image;

[0011] Among them, the adopted multi-scale digital network model uses a generative adversarial network model based on a multi-discriminator architecture; the multi-discriminator architecture includes a content discriminator for promoting the separation of content features and makeup styles, a global discriminator for evaluating the face region and identifying the quality and realism of the makeup transfer of the facial makeup image, and a local discriminator for local makeup detection and correction of the makeup transfer image.

[0012] As a further technical limitation, the local discriminator uses a local loss function for constraining the color distribution consistency of the images before and after makeup transfer in the local region to ensure the consistency of the local makeup styles of the images before and after makeup transfer.

[0013] Furthermore, with the goal of minimizing the objective function of the multi-scale digital network model including the local loss function, combined with the detection and recognition of the global discriminator and the local discriminator, optimize and adjust the process of makeup transfer of the facial makeup image.

[0014] As a further technical limitation, a content encoder is used to extract the content features of the facial makeup image, and a makeup style encoder is used to extract the makeup styles of the facial makeup image; the makeup style encoder uses a convolutional neural network with a hierarchical structure for capturing the makeup styles related to makeup in the facial makeup image.

[0015] As a further technical limitation, the face region is segmented into several local face blocks, and a global discriminator adopting a multi-scale discriminant network structure is used to judge the makeup style of the face region. A local discriminator is used to detect and correct the face makeup of the local face blocks, and the effect of makeup transfer is evaluated according to the judgment and detection results of the global discriminator and the local discriminator.

[0016] As a further technical limitation, a generator introducing a face parsing map is used to segment the obtained face makeup image into different semantic regions. According to the obtained semantic regions, the background region of the face makeup image is obtained to complete the extraction of the background region. The face parsing map is used to extract the pixel of the face region of the makeup transfer image to obtain the face region of the makeup transfer image.

[0017] According to some embodiments, the second solution of the present invention provides a multi-scale digital makeup transfer system, adopting the following technical solution:

[0018] A multi-scale digital makeup transfer system includes:

[0019] An acquisition module configured to acquire different face makeup images;

[0020] A transfer module configured to perform makeup transfer on the face makeup image by using a multi-scale digital network model to obtain a face makeup image after makeup transfer. Specifically, in the process of performing makeup transfer on the face makeup image, the content features, makeup style and background region of the obtained different face makeup images are extracted. Makeup transfer is performed according to the extracted content features of the face makeup image to obtain a makeup transfer image, and the transferred makeup style is different from the makeup style of the face makeup image to be subjected to makeup transfer. The face region of the obtained makeup transfer image is extracted. The extracted background region and face region are fused to obtain a face makeup image after makeup transfer, realizing the makeup transfer of the face makeup image.

[0021] Among them, the adopted multi-scale digital network model adopts a generative adversarial network model based on a multi-discriminator architecture. The multi-discriminator architecture includes a content discriminator for promoting the separation of content features and makeup styles, a global discriminator for evaluating the face region and identifying the quality and realism of the makeup transfer of the face makeup image, and a local discriminator for local makeup detection and correction of the makeup transfer image.

[0022] According to some embodiments, the third solution of the present invention provides a computer-readable storage medium, adopting the following technical solution:

[0023] A computer-readable storage medium, on which a program is stored, and when the program is executed by a processor, the steps in the multi-scale digital makeup transfer method described in the first solution of the present invention are implemented.

[0024] According to some embodiments, the fourth solution of the present invention provides an electronic device, adopting the following technical solution:

[0025] An electronic device includes a memory, a processor, and a program stored on the memory and running on the processor. When the processor executes the program, it implements the steps in the multi-scale digital makeup migration method described in the first solution of the present invention.

[0026] According to some embodiments, the fifth solution of the present invention provides a computer program product, adopting the following technical solution:

[0027] A computer program product includes software code, and the program in the software code executes the steps in the multi-scale digital makeup migration method described in the first solution of the present invention.

[0028] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0029] The present invention introduces a multi-discriminator combination strategy into the multi-scale digital network model and designs a new multi-discriminator architecture, which can effectively improve the performance and effect of makeup migration of the multi-scale digital network model; the multi-discriminator architecture consists of a content discriminator for judging the authenticity of face content information, two global discriminators for judging the authenticity of the whole face, and ten local discriminators for judging the authenticity of local face regions.

[0030] The present invention proposes a new local loss function, which is used to constrain the color distribution consistency between the image after makeup migration and the original made-up image in the local area; introducing this function into the local discriminator can ensure that the image after makeup migration is consistent with the original image in the local makeup style, and can guide the generator to better learn and maintain the details and features of each area, generating a more realistic makeup migration result, thereby improving the overall quality of the makeup migration image.

[0031] To avoid changes in the background information of the image during the makeup migration process and cause adverse interference to the final facial makeup effect, the present invention proposes the idea of separately processing the face region, and introduces a face parsing map into both the generator and the discriminator to separate the face region and the background region of the image. When the discriminator evaluates the face image, it only evaluates the face region, thereby eliminating the influence of background information on the facial makeup migration effect; when the generator generates an image, the face parsing map is used to extract the face region of the network layer output result and the background region of the original image respectively, and then the two parts are combined into the final generated image, thereby ensuring the consistency of the background of the face makeup image after makeup migration with the original image. The present invention only considers and focuses on facial features and details during the makeup migration process, and can generate a more natural and realistic makeup migration effect. Description of the Drawings

[0032] The attached drawings forming a part of this embodiment are used to provide a further understanding of this embodiment. The schematic embodiments and descriptions thereof of this embodiment are used to explain this embodiment and do not constitute an improper limitation to this embodiment.

[0033] Figure 1 It is the architecture diagram of the multi-scale digital network model in the first embodiment of the present invention;

[0034] Figure 2 It is the structure diagram of the global discriminator in the first embodiment of the present invention;

[0035] Figure 3 It is the structure diagram of the local discriminator in the first embodiment of the present invention;

[0036] Figure 4 It is the schematic diagram of the experimental comparison results of the MT dataset in the first embodiment of the present invention;

[0037] Figure 5 It is the schematic diagram of the experimental comparison results of the Makeup dataset in the first embodiment of the present invention;

[0038] Figure 6 It is the schematic diagram of the experimental comparison results of the makeup removal of the MT dataset in the first embodiment of the present invention;

[0039] Figure 7 It is the schematic diagram of the experimental comparison results of the makeup removal of the Makeup dataset in the first embodiment of the present invention. Detailed Embodiments

[0040] The present invention will be further described below in conjunction with the drawings and embodiments.

[0041] It should be noted that the following detailed descriptions are all exemplary and are intended to provide further descriptions of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the technical field to which this application belongs.

[0042] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular forms are also intended to include the plural forms. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0043] In the present invention, terms such as "upper", "lower", "left", "right", "front", "rear", "vertical", "horizontal", "side", "bottom", etc. indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings. They are only relational terms determined for the convenience of describing the structural relationships of various components or elements of the present invention, and do not specifically refer to any component or element in the present invention, and should not be construed as a limitation to the present invention.

[0044] In the case of no conflict, the embodiments in the present invention and the features in the embodiments can be combined with each other.

[0045] Embodiment 1

[0046] Embodiment 1 of the present invention introduces a multi-scale digital makeup transfer method.

[0047] This embodiment uses a multi-scale digital network model for makeup transfer; that is, a generative adversarial network model based on a multi-discriminator architecture including a content discriminator, a global discriminator, and a local discriminator is used; among them, the content discriminator, the global discriminator, and the local discriminator are respectively used for separating content information and makeup style information, evaluating the overall facial authenticity at a large scale, and evaluating the authenticity of the local facial area at a small scale. The combination of the three types of discriminators can enable the generator to generate a more realistic and natural facial makeup transfer effect; a local loss function is constructed to constrain the color distribution consistency of the makeup transfer image and the original made-up image in the local area, and this loss function is used for the local discriminator, which helps the network to better transfer the details of the makeup; a face parsing map is introduced into the generator and the discriminator, which is used to extract the face area for separate processing, so as to reduce the adverse impact of the background on the facial makeup transfer effect, and ensure that the finally generated facial makeup image after makeup transfer is consistent with the background of the same identity image in terms of the background.

[0048] This embodiment is introduced in detail with two different makeup looks as examples.

[0049] Let A and B be two different domains of facial makeup images, where A represents the domain of makeup-free images, B represents the domain of made-up images, and both A and B contain images with different identities and makeup looks, and there are no paired images of the same identity before and after makeup. Assuming that the makeup-free image is a special case of the made-up image, based on this assumption, removing makeup and applying makeup can be regarded as the same problem. Specifically, when four given images are input: the makeup-free image I s ∈A, the made-up image I m ∈B, the face parsing map M s of the makeup-free image, and the face parsing map M m of the made-up image, through the network model in this embodiment, the made-up image and the makeup-removed image is an image that simultaneously has Is Identity and I m makeup style images, and Background information and I s Consistent, that is, to achieve I s Makeup effect. It is a card that has both I m Identity and I s images with makeup styles (i.e., no makeup), and Background information and I m Consistent, that is, achieved m The effect of makeup removal. By learning the mapping relationship between the two domains: To realize the makeup application and removal of facial makeup images.

[0050] This embodiment adopts Figure 1 The multi-scale digital network model shown includes a makeup style encoder, a content encoder, a generator, a content discriminator, a global discriminator, and a local discriminator.

[0051] (1) Content Encoder

[0052] The content encoder is used to transform the input makeup-free image I s ∈A and makeup image I m ∈B, which captures the content features of the image such as facial structure and expression without considering the makeup style.

[0053] The content encoder in this embodiment is composed of multiple convolutional layers, activation function layers and normalization layers, and is used to capture content information from the face image. The captured content information includes the contour of the face, the shape and position of the eyes, nose, mouth and other parts. The content encoder can also extract higher-level abstract features, such as facial expressions, postures, gender, age and other information. The content encoder plays a key role in the entire makeup transfer process. It converts the input face image into a potential content space representation, so that the generator can use the content space representation to maintain the consistency of the face identity when performing makeup transfer.

[0054] Extract content features and The process is:

[0055] (2) Makeup style encoder

[0056] The makeup style encoder is used to extract the input image I s ∈A and I m The latent features of makeup style ∈B map the image to the attribute space representing the variation of makeup style.

[0057] The makeup style encoder in this embodiment is used to capture makeup-related information in an image. The design and training of the makeup style encoder are crucial for achieving accurate makeup transfer effects. In this embodiment, a makeup style encoder with a Conv-Relu-Conv-BN structure is used to extract potential feature representations of makeup styles such as eye makeup, lip makeup, foundation, and blush from the image. Among them, the first convolutional layer in this structure is responsible for extracting basic features, and the second convolutional layer can further extract more complex features based on these basic features. The hierarchical structure enables the makeup style encoder to learn richer and more diverse feature representations, thereby better capturing the makeup style of the input data. The output of the makeup style encoder is input into the generator to guide the generator to generate images with specific makeup styles. Extracting makeup style features and The process of

[0058]

[0059] (3) Generator

[0060] The generator takes the original image I s and I m and its corresponding face parsing map M s and M m , as well as the face features extracted by the content encoder and the makeup style encoder and as inputs. After fusing the features extracted by the two encoders, the new features are input into the network layer to obtain the output result. The face region information is extracted from the output result using the face parsing map, and the background region information is extracted from the original image. The two parts of information extracted are combined to generate the makeup-on image and the makeup-off image

[0061] The face regions (i.e., removing background information) are respectively extracted from the generated images and real images of the generator and used as inputs to the global discriminator, enabling the global discriminator to evaluate whether the input face regions belong to the same makeup style. At the same time, the face region with background information removed is segmented into multiple local blocks of the face, and these local blocks are used as inputs to the local discriminator, enabling the local discriminator to evaluate whether the corresponding face local blocks between the input generated image and the real image belong to the same makeup style. Through the combined action of the global discriminator and the local discriminator, the network can more accurately evaluate the makeup style of the image, thereby generating more realistic makeup transfer results.

[0062] As an image processing technology, the face parsing map divides the face image into different semantic regions, such as skin, eyes, mouth, hair, etc. In this embodiment, the face parsing map is introduced into the generator to separate the face region and the background region in the image and process them separately, so as to avoid the inconsistency between the background of the generated face makeup image after makeup transfer and the background of the original image.

[0063] Specifically, the face parsing map is used to classify the pixels in the face image into two categories: face region pixels and background region pixels; among them, the face region pixels include pixels in the makeup area such as eyebrows, lips, skin, nose, etc., while the background region pixels include pixels in the non-makeup area such as background, hair, teeth, eyeballs, etc. By classifying the pixels in the image using the face parsing map, the model extracts the face region pixels from the image generated by the generator and the background region pixels from the original image, and combines the pixels extracted from different images to generate a new image with only the face makeup changed and the background unchanged, that is, the face makeup image after makeup transfer is obtained.

[0064] In this embodiment, the generator adopted not only receives the content features extracted by the encoder and the makeup style features but also simultaneously receives the original image I s 、I m and its corresponding face parsing map M s 、M m as inputs. In this embodiment, the generator G is divided into two branches: makeup application and makeup removal. Each branch consists of a feature fusion module, three Conv-BN-Relu layers, two upsampling layers and a Conv-Tanh layer. An upsampling operation is performed on the output features of the first two Conv-BN-Relu layers, which helps to maintain the spatial information and structure of the input image, especially crucial in retaining details such as object edges and textures, so that the generated image is more realistic. When all the inputs enter the generator, the makeup-free image I s 、the makeup-free face parsing map M s and the makeup-free image content features extracted by the encoder and the makeup image makeup style features are sent to the makeup application branch of the generator. In the feature fusion module, the content features and the makeup style features are fused to form new features The fused new features are input into the network composed of Conv-BN-Relu layer, upsampling layer and Conv-Tanh layer to obtain the output result. The makeup-free face parsing map M s is used to extract the makeup-free image I sThe pixel of the background area and the pixel of the face area of the output result are combined, and the two parts of pixels are combined to generate a final makeup image with the identity information of the makeup-free image, the background information of the makeup-free image, and the makeup style of the makeup image Similarly, the makeup image I m , the makeup face parsing map M m , the content features of the makeup image and the makeup style features of the makeup-free image are sent to the makeup removal branch of the generator. After being processed by the feature fusion module, the network layer, and the classification and combination of pixels by the face parsing map, a makeup removal image with the identity information of the makeup image, the background information of the makeup image, and the makeup style of the makeup-free image is generated During the makeup application and makeup removal processes, the introduction of the face parsing map can effectively distinguish whether the pixel points in the image belong to the face area or the background area, facilitating the separate processing of pixel points in different areas of different images, so that the finally generated makeup image and makeup removal image can both maintain the same background as the original input image of the same identity

[0065] The process of generating an image by the generator in this embodiment is as follows

[0066] (4) Content discriminator

[0067] The content discriminator is used to judge whether the information contained in the feature vector output by the content encoder is content information that is common between domains or attribute information of a specific domain (that is, content feature information unrelated to makeup or makeup style information).

[0068] The content discriminator and the content encoder form an adversarial training to judge whether the information output by the content encoder is content information that is common between domains or attribute information of a specific domain. In this embodiment, the content discriminator consists of 4 Conv-IN-LeakyRelu layers and one Conv-Sigmoid layer, using the content features extracted by the content encoder from the makeup-free image and the makeup image as the input. By evaluating this information, it is judged whether it is information related to the makeup style or content information unrelated to makeup; by constructing a content adversarial loss function between the content encoder and the content discriminator, the separation of content information and makeup style information is effectively promoted during the encoding process

[0069] (5) Global discriminator

[0070] In the generative adversarial network, the discriminator plays a crucial role in accurately distinguishing real images and generated images, thereby driving the generator to create more realistic results. To improve the accuracy of the model, two global discriminators are introduced in this embodiment; among them, the global discriminator D AFocus on evaluating the authenticity and realism of makeup-removing images and makeup-free images, the global discriminator D B is used to evaluate the authenticity and realism of makeup-application images and made-up images. The global discriminator D A and D B adopt the same network structure, Figure 2 for D A is the network structure diagram.

[0071] The global discriminator in this embodiment adopts a multi-scale discrimination network structure. For the images at the original scale and the features downsampled using the average pooling layer, they respectively pass through four Conv-IN-LeakyRelu layers and one Conv-sigmoid layer, and finally obtain the evaluation results of the discriminator on real images and generated images. Features at different scales represent different levels of information in the image. By discriminating on face features at different scales, the discriminator can pay attention to more comprehensive information and improve the ability to understand the content of face images.

[0072] For the global discriminator D A , before inputting the makeup-free image I s and the makeup-removing image into the discriminator, this embodiment uses a face parsing map to classify the image pixels, fills the background with black to remove the background area information in the image, and extracts the face regions s in the makeup-free image I and the makeup-removing image and It should be noted that in this embodiment, the eyeballs and teeth do not belong to the makeup area, so these two parts are excluded when extracting the face region. In this way, only the makeup-free image of the face region and the makeup-removing image of the face region are sent to the discriminator for evaluation; effectively preventing the adverse interference of background information on the makeup application and removal processes, thus ensuring that the discriminator can focus on evaluating the makeup style on the face and making the generated images more accurate and realistic.

[0073] The processing flow of the global discriminator D B and D A is the same, which will not be elaborated in this embodiment.

[0074] (6) Local discriminator

[0075] In this embodiment, 10 local discriminators are set up to evaluate the authenticity of local regions of the human face during makeup transfer. The face region with background information removed is divided into 10 local blocks using facial key points. After splicing, these local blocks contain complete facial details and features. One local discriminator is set for each facial local block to focus on evaluating the corresponding local region of the human face. The sum of the regions evaluated by these 10 local discriminators constitutes the entire human face region. This fine-grained facial region division and local evaluation strategy enable the model to capture and evaluate each part of the human face more accurately, further improving the quality and realism of the generated images.

[0076] The 10 local discriminators in this embodiment adopt the same network structure, which is composed of a combination of 6 convolutional layers, normalization layers, and activation function layers (Conv-IN-LeakyRelu) and a combination of 1 convolutional layer and activation function layer (Conv-Sigmoid). The network structure diagram of the local discriminator is as Figure 3 shown.

[0077] In this embodiment, the made-up image I m is deformed and fused onto the makeup-free image I s to generate the synthetic made-up image W m . The facial regions in the synthetic made-up image W m , the made-up image , and the made-up image I m are extracted using the face parsing map. Then, using facial key points, the facial regions of the synthetic made-up image, the made-up image facial region , and the made-up image facial region and the made-up image facial region are segmented into local blocks. The made-up facial local blocks of I m and and the made-up facial local blocks of are input into the local discriminator corresponding to this local block, and the discriminator is made to judge this pair of local blocks with different makeup styles as negative. The made-up facial local blocks of I m and and the synthetic made-up facial local blocks of W m are input into , and the discriminator is made to judge this pair of local blocks with the same makeup style as positive. The local discriminator not only maintains the integrity of the makeup color and complex makeup patterns during the makeup process but also, to a certain extent, keeps the identity information of the human face unchanged.

[0078] In this embodiment, the local discriminator and the global discriminator are combined, enabling the model to not only evaluate the authenticity of the entire face but also that of local regions of the face. Since the division of the face regions is based on key points, the sum of the regions evaluated by multiple local discriminators can cover the entire face without missing edge details. During the transfer process of complex makeup, the global discriminator focuses on overall consistency, while the local discriminators focus on detailed features. The combination of the two can better capture the details of the makeup and provide more comprehensive feedback signals, contributing to improving the quality of makeup transfer in terms of both coarse-grained overall makeup and fine-grained local details.

[0079] In this embodiment, the content discriminator D c is used to distinguish the content features extracted from the makeup-free image and the made-up image, enabling the content encoder to focus more on content information unrelated to makeup during data encoding and reducing its dependence on makeup style information. The makeup style encoder, on the other hand, encodes the remaining specific makeup style information. The content adversarial loss function is where E represents the expectation operation on the image.

[0080] The global discriminator takes as input an image with background information removed, i.e., an image that only retains the region of the complete face. D A is used to distinguish the face region of the makeup-removed image from that of the makeup-free image, and D B is used to distinguish the face region of the makeup-applied image from that of the made-up image. The global generative adversarial loss is

[0081] In this embodiment, the extracted face region is segmented into 10 local face blocks, and a local discriminator is set for each local face block. The local discriminator takes the local face block as input and determines that the local block pairs of the made-up image I m and the synthesized makeup-applied image W m are positive pairs, i.e., they are considered to have the same makeup style; it determines that the local block pairs of the made-up image I and the makeup-applied image generated by the generator m are negative pairs, i.e., they are considered to have different makeup styles. The local generative adversarial loss function is

[0082]

[0083] ​​In this embodiment, it is desired that the makeup style features extracted by the makeup style encoder be as close as possible to the prior Gaussian distribution. Therefore, the KL (Kullback-Leibler) divergence loss is used for the makeup style encoder. Here, the KL divergence loss is used to measure the difference between the distribution of makeup style features and the prior Gaussian distribution, preventing the distribution of makeup style features from being too discrete or concentrated in a certain local area, thus ensuring that the makeup style features are more stable and controllable in the entire feature space. The KL divergence loss function is: where P(x) represents the probability density function of the distribution of makeup style features, and Q(x) represents the probability density function of the Gaussian distribution.

[0084] The reconstruction loss is a combination of the cycle consistency loss and the self-reconstruction loss. The content information s and the makeup style information extracted from the no-makeup image I are input into the generator, and the no-makeup image will be regenerated. Similarly, the content information m and the makeup style information of the made-up image I are input into the generator, and the made-up image will also be regenerated. Here, the difference between I s and , and the difference between I m and is called the self-reconstruction loss. For the makeup-on image and the makeup-removal image generated by the generator, the content information and the makeup style information are re-extracted, and then they are swapped and fused and input into the generator to produce a cyclic no-makeup image s consistent with I and a cyclic made-up image m consistent with I The difference between I s and , and the difference between I m and is the cycle consistency loss. Therefore, the reconstruction loss function is

[0085]

[0086] In this embodiment, a smooth makeup-removal loss is used to constrain the generator to produce a better makeup-removal image. The face after makeup removal should have no high-frequency details but be smoother. The Laplace filter f is sequentially used for the divided local face blocks, and it is desired that the resulting result be closer to 0. Since the facial features included in the local face blocks are different, separate weights ω are set for each local face block of the makeup-removal image i; The smooth makeup removal loss function is

[0087] In this embodiment, when calculating the loss of the local face region, based on the face parsing map, the background, hair, eyeballs, eyebrows, and teeth regions in the face image are masked, and the remaining face region is used as the region affected by makeup transfer and divided into 10 local blocks; the eyebrow region is removed here because it is desired that the color rather than the shape of the eyebrows is transferred. If the eyebrow region is retained and the local loss function is calculated, the trace of the original made-up eyebrow shape will be left in the final face makeup image after makeup transfer, which does not match the expected makeup result. Therefore, the eyebrows are masked when calculating the local loss function, and other global loss functions are relied on to achieve the constraint on the eyebrow color transfer effect. The face makeup style mainly depends on the color distribution. Therefore, the makeup application image and the made-up image I m should have a similar color distribution in the makeup region.

[0088] To take care of more makeup details and produce a better makeup transfer effect during the makeup transfer process, this embodiment imposes a constraint on the color distribution consistency of each local block of the face. Specifically, first, the face region of the makeup-free image is extracted using the face parsing map the face region of the made-up image and the face region of the synthesized makeup application image Secondly, the face regions are each segmented using their respective key points to obtain 10 local blocks, and the sum of the local blocks contains all the information of each face region; thirdly, the local blocks of the makeup-free face and the corresponding local blocks on the made-up face are subjected to histogram matching. The histogram of the local block of the makeup-free face is adjusted to be similar to the histogram of the local block of the made-up face . The local block of the makeup-free face with the adjusted histogram is called the histogram-matched local block, denoted as This histogram-matched local block retains both the identity information of the local block of the makeup-free face and has the same color distribution as the local block of the made-up face ; finally, the L1 norm is used to calculate the difference between the histogram-matched local block and the local block of the makeup-applied face . The local loss function plays a key role in the color transfer of extreme makeup. The local loss function is where HM represents histogram matching.

[0089] In summary, the total loss function can be obtained as where λ content, λ adv , λ local , λ kl , λ rec , λ smooth and λ region_hm are the weights of each loss function.

[0090] Example analysis

[0091] In this embodiment, the Adam algorithm is used to optimize all modules of the model, and the initial learning rate is set to 0.0001. The model is trained for 700 epochs with a batch size of 1, and the weight parameters λ content , λ adv , λ local , λ kl , λ rec , λ smooth and λ region_hm are set to 1, 1, 8, 0.01, 8, 15, and 1.5 respectively.

[0092] In this embodiment, two datasets, MT and Makeup, are used. The MT dataset consists of 1115 makeup-free images and 2719 made-up images, and the images therein include different races, poses, expressions, backgrounds, etc. Although the MT dataset contains many makeup styles, these makeup styles all belong to the light makeup category. The Makeup dataset contains a total of 334 makeup-free images and 355 made-up images; it includes both light makeup images and images with extreme makeup; these images present diverse makeup colors and styles.

[0093] 300 makeup-free images and 300 made-up images are selected from the Makeup dataset to construct a training set, and it is ensured that the images with extreme makeup are not all included in the training set. This training set is used to train the multi-scale digital network model proposed in this embodiment so that the model has good makeup transfer performance that can adapt to both light makeup and extreme makeup simultaneously.

[0094] Two test datasets are constructed; 300 makeup-free images and 200 made-up images are randomly selected from the MT dataset to construct the first test dataset, called the MT test dataset. After removing the training data from the Makeup dataset, the remaining images are used to construct the second test dataset, called the Makeup test dataset.

[0095] The method proposed in this example is compared with several advanced makeup transfer methods, including BeautyGAN, SCGAN, BeautyREC, and CPM. In order to more accurately evaluate the performance of each method, the result images generated by all methods are aligned to the same resolution (256*256). This can eliminate the visual differences caused by different image resolutions, making the comparison more comparable and fair.

[0096] like Figure 4 The figure shows the comparison results of the method adopted in this embodiment and other comparison methods on the MT dataset, which is mainly light makeup style; among them, the first row is the images without makeup, the second row is the images with makeup, and the third to seventh rows are the makeup migration results obtained by BeautyGAN, SCGAN, BeautyREC, CPM and the method of this embodiment respectively.

[0097] Depend on Figure 4 It can be seen that the makeup transfer effect obtained by the BeautyGAN method is not ideal, and the face appears black. The makeup transferred by the SCGAN method changes color, such as the lip makeup shown in the fourth row, column (b) and column (f), and the eye makeup and lip makeup shown in column (e). And the image generated by this method is overall blurry, and the hair and background are seriously distorted. Compared with the first two methods, the BeautyREC and CPM methods perform better. However, the BeautyREC method only transfers the color of the lower lip and ignores the upper lip when migrating lip makeup. This phenomenon can be seen in the results of the BeautyREC method. The lip makeup color of the makeup image generated by the CPM method is lighter than the lip makeup color of the corresponding makeup image, as shown in the CPM method result figure. Compared with the above methods, the method proposed in this embodiment not only realizes the makeup transfer well, ensures the accuracy of the makeup color and details, but also maintains the background of the original image well. Compared with other methods, the makeup transfer effect obtained in this embodiment is more natural and realistic.

[0098] like Figure 5 The figure shows the comparison results between the method adopted in this embodiment and the BeautyGAN, SCGAN, BeautyREC and CPM methods on the Makeup dataset; some extreme makeups in the Makeup dataset are selected to demonstrate the superiority of the method adopted in this embodiment in extreme makeup migration; wherein, the first row is the images without makeup, the second row is the images with makeup, and the third to seventh rows are the makeup migration results obtained by BeautyGAN, SCGAN, BeautyREC, CPM and the method of this embodiment respectively.

[0099] Depend on Figure 5It can be seen that the makeup transfer effect obtained by the BeautyGAN method is very poor, and the generated face has serious darkening. The generation result of SCGAN only realizes the transfer of lip color, and none of the other facial special effect makeup is transferred, and the background and hair have color changes and blurring phenomena. The BeautyREC method also only realizes the transfer of lip color and has no transfer effect on facial special effect makeup. Some results of the CPM method realize a certain degree of facial special effect makeup transfer, but the makeup pattern transfer is incomplete and inaccurate. For example, the content in the (a) column of the sixth row shows that there is a color overstep phenomenon after makeup transfer. In addition, there is an obvious boundary between the makeup color and the natural skin color on the forehead of multiple makeup images generated by this method, as shown in the figure. Among the compared methods, BeautyGAN, SCGAN, and BeautyREC cannot realize the transfer of complex facial makeup, and the CPM method can realize the transfer of some complex facial makeup, but there are problems such as color overstep of the transferred makeup and background change. The method adopted in this embodiment shows better effects for the transfer of extreme makeup and background preservation. It can not only accurately transfer the shape and color of the special effect makeup pattern but also well preserve the original hair color and background of the image.

[0100] The method adopted in this embodiment can also realize the makeup removal function for made-up images. Since the BeautyREC and CPM methods cannot realize makeup removal, through Figure 6 and Figure 7 comparison, the makeup removal effects of the method adopted in this embodiment and the BeautyGAN and SCGAN methods on the MT dataset and Makeup dataset can be obtained. Among them, the first column is the made-up image, the second column is the makeup removal result of the BeautyGAN method, the third column is the makeup removal result of the SCGAN method, and the fourth column is the makeup removal result of the method adopted in this embodiment. As Figure 6 shown in the makeup removal effect of light makeup. It can be seen that in the makeup removal result of BeautyGAN, the lip makeup of some images is successfully removed, but the other makeup is basically not removed, and there is a problem of facial darkening. The makeup removal images obtained by the SCGAN method show an overall whitish effect, and the image clarity is low. In addition, the background colors of the makeup removal images obtained by these two methods have changed. In contrast, the method adopted in this embodiment has achieved a better makeup removal effect. Not only the lip makeup and eye makeup are effectively removed, but also the facial skin color remains natural. In addition, the background in the makeup removal result of the method adopted in this embodiment is well preserved and has no obvious change compared with the original image. This shows that this embodiment can not only effectively remove makeup but also maintain the overall tone of the image and the consistency of the background, achieving a more natural and clear makeup removal effect.

[0101] As Figure 7The figure shows a comparison diagram of the makeup removal effects of extreme makeup in the Makeup dataset. The BeautyGAN method hardly removes the special effects makeup, and there is still a phenomenon of facial darkening. The SCGAN method has a great impact on the eyes and the background during the makeup removal process, resulting in eye deformation, background blurring and changes. The method adopted in this embodiment has a better makeup removal effect for extreme makeup, less color residue, and the most complete background information retained.

[0102] To quantitatively compare the identity preservation effects of the method adopted in this embodiment and other methods during makeup application and removal, in this embodiment, ArcFace is used to calculate the facial similarity between the makeup applied images and the makeup-free images, as well as the facial similarity between the makeup removed images and the makeup-on images on the MT dataset and the Makeup dataset respectively. The larger the value of this index, the better the identity characteristics of the human face are preserved after makeup application or removal. For the MT dataset, in this embodiment, 100 makeup-free images and 100 makeup-on images in the test set are randomly selected to generate makeup applied images and makeup removed images respectively; for the Makeup dataset, in this embodiment, the entire test set is used, that is, 34 makeup-free images and 55 makeup-on images are used to generate makeup applied and makeup removed images respectively. The results calculated on the MT dataset are shown in Table 1. It can be seen that whether it is makeup application or removal, the ArcFace value in this embodiment is the highest, indicating that the method adopted in this embodiment can well preserve the identity characteristics during makeup application and removal.

[0103] Table 1 ArcFace values calculated by different methods on the MT dataset (↑)

[0104]

[0105] The results calculated on the Makeup dataset are shown in Table 2. It can be seen that for makeup removal, the ArcFace value of the method adopted in this embodiment is the highest. For makeup application, the ArcFace value of the method in this embodiment is the second highest. The ArcFace value of the BeautyREC method is the highest because there are some human face images with complex makeup on the face in this dataset, and the BeautyREC method can hardly migrate these makeup (as shown in Figure 7 ), so the makeup applied images obtained are very similar to the original makeup-free images, resulting in a higher ArcFace value. Note that the BeautyREC and CPM methods cannot achieve the makeup removal function, so the ArcFace values when they remove makeup are not calculated in Table 1 and Table 2. In summary, compared with other methods, the method in this embodiment has a better effect of preserving identity characteristics during makeup application and removal.

[0106] Table 2 ArcFace values calculated by different methods on the Makeup dataset (↑)

[0107]

[0108] To quantitatively compare the similarity of the makeup and the similarity of the makeup removal between the method adopted in this embodiment and other methods, in this embodiment, the FID is used on the MT image and the Makeup image to calculate the similarity between the makeup application image and the made-up image, and the similarity between the makeup removal image and the makeup-free image. The smaller the FID value, the greater the similarity. The calculation results on the MT dataset are shown in Table 3, and the calculation results on the Makeup dataset are shown in Table 4. It can be seen that whether it is makeup application or makeup removal, the FID value of the method in this embodiment is the highest, indicating that the makeup of the makeup application image generated by the method in this embodiment is closer to the makeup of the corresponding made-up image, and the makeup of the makeup removal image generated is closer to the makeup of the corresponding makeup-free image.

[0109] Table 3 FID values calculated by different methods on the MT dataset (↓)

[0110]

[0111] Table 4 FID values calculated by different methods on the Makeup dataset (↓)

[0112]

[0113] This embodiment uses a generative adversarial network framework with a combination of multiple discriminators for digital makeup transfer work; the collaborative work of multiple discriminators enables the framework to effectively separate the face content information and the makeup style information, and achieve a good transfer effect from the large-scale overall makeup to the small-scale local makeup details; among them, the 10 proposed local discriminators play a key role in maintaining the face identity information and the accuracy of the makeup color; the proposed local loss function plays an important role in the transfer of the color and texture of the makeup style, which can ensure that the generated makeup transfer image is consistent with the target image in terms of makeup style, and can also fully transfer extreme makeup styles; the face parsing map introduced in the generator and the discriminator can separate the background and the face area, effectively solve the problem that the image background information changes during the makeup application and makeup removal processes and the adverse interference of the background information on the transfer effect, making the final face makeup image after makeup transfer more natural visually and the makeup transfer effect more accurate.

[0114] Embodiment 2

[0115] Embodiment 2 of the present invention introduces a multi-scale digital makeup transfer system.

[0116] A multi-scale digital makeup transfer system includes:

[0117] An acquisition module configured to acquire different face makeup images;

[0118] A migration module, which is configured to perform makeup migration on a face makeup image by using a multi-scale digital network model to obtain a face makeup image after makeup migration; specifically, during the process of performing makeup migration on a face makeup image, the content features, makeup styles, and background regions of the obtained different face makeup images are extracted; makeup migration is performed according to the extracted content features of the face makeup image to obtain a makeup migration image, and the makeup style to be migrated is different from the makeup style of the face makeup image to be subjected to makeup migration; the face region of the obtained makeup migration image is extracted; the extracted background region and face region are fused to obtain a face makeup image after makeup migration, thereby realizing the makeup migration of the face makeup image.

[0119] Among them, the adopted multi-scale digital network model adopts a generative adversarial network model based on a multi-discriminator architecture; the multi-discriminator architecture includes a content discriminator for promoting the separation of content features and makeup styles, a global discriminator for evaluating the face region and identifying the quality and fidelity of the makeup migration of the face makeup image, and a local discriminator for local makeup detection and correction of the makeup migration image.

[0120] The detailed steps are the same as those of the multi-scale digital makeup migration method provided in Embodiment 1 and will not be elaborated here.

[0121] Embodiment 3

[0122] Embodiment 3 of the present invention provides a computer-readable storage medium.

[0123] A computer-readable storage medium, on which a program is stored, and when the program is executed by a processor, the steps in the multi-scale digital makeup migration method as described in Embodiment 1 of the present invention are implemented.

[0124] The detailed steps are the same as those of the multi-scale digital makeup migration method provided in Embodiment 1 and will not be elaborated here.

[0125] Embodiment 4

[0126] Embodiment 4 of the present invention provides an electronic device.

[0127] An electronic device includes a memory, a processor, and a program stored on the memory and running on the processor. When the processor executes the program, the steps in the multi-scale digital makeup migration method as described in Embodiment 1 of the present invention are implemented.

[0128] The detailed steps are the same as those of the multi-scale digital makeup migration method provided in Embodiment 1 and will not be elaborated here.

[0129] Embodiment 5

[0130] Embodiment 5 of the present invention provides a computer program product.

[0131] A computer program product includes software code, and the program in the software code executes the steps in the multi-scale digital makeup migration method as described in Embodiment 1 of the present invention.

[0132] The detailed steps are the same as those of the multi-scale digital makeup migration method provided in Embodiment 1 and will not be elaborated here.

[0133] The above are only the preferred embodiments of this embodiment and are not used to limit this embodiment. For those skilled in the art, various changes and modifications can be made to this embodiment. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this embodiment shall be included within the protection scope of this embodiment.

Claims

1. A multi-scale digital makeup migration method, characterized in that: include: Obtain images of different facial makeup; A multi-scale digital network model is used to transfer makeup from a facial makeup image to obtain a facial makeup image after makeup transfer. Specifically, in the process of performing makeup transfer of facial makeup images, content features, makeup styles and background areas of different acquired facial makeup images are extracted; makeup transfer is performed according to the content features of the extracted facial makeup images to obtain a makeup transfer image, wherein the migrated makeup style is different from the makeup style of the facial makeup image to be transferred; a facial area of ​​the obtained makeup transfer image is extracted; the extracted background area and the facial area are fused to obtain a facial makeup image after makeup transfer, thereby realizing makeup transfer of the facial makeup image; The multi-scale digital network model used adopts a generative adversarial network model based on a multi-discriminator architecture; the multi-discriminator architecture includes a content discriminator for facilitating the separation of content features and makeup styles, a global discriminator for evaluating the face area and identifying the quality and realism of makeup transfer of face makeup images, and a local discriminator for local makeup detection and correction of makeup transfer images; The content encoder is used to extract content feature representations from the input images without makeup and with makeup, which captures the content features in the image without considering the makeup style; The generator takes the original image and its corresponding face analysis map, as well as the facial features extracted by the content encoder and the makeup style encoder as input; Before inputting the makeup-free and makeup-removed images into the discriminator, the face parsing map is used to classify the image pixels; The specific steps of the local discriminator are: deforming and fusing the makeup image into the non-makeup image I s On the top, a synthetic makeup image W is generated m , using the face analysis graph to extract the synthetic makeup image W m , Makeup images And makeup image I m The facial area in the extracted synthetic makeup image is then extracted using the facial key points. Makeup image face area And the face area of ​​the makeup image Perform local block segmentation on I m Partial block of the face with makeup and Partial blocks of makeup face Input to the local discriminator corresponding to the local block In the example, let the discriminator judge the pair of local blocks with different makeup styles as negative, and I m Partial block of the face with makeup and W m Synthetic makeup face local block Input to In , let the discriminator judge the pair of local blocks with the same makeup style as positive.

2. A multi-scale digital makeup migration method as claimed in claim 1, characterized in that: The local discriminator adopts a local loss function for constraining the color distribution consistency of the image before and after makeup transfer in the local area, thereby ensuring the consistency of the local makeup style of the image before and after makeup transfer.

3. A multi-scale digital makeup migration method as claimed in claim 2, characterized in that: Taking the minimization of the objective function of the multi-scale digital network model including the local loss function as the goal, the detection and recognition of the global discriminator and the local discriminator are combined to optimize and adjust the makeup transfer process of the facial makeup image.

4. A multi-scale digital makeup migration method as claimed in claim 1, characterized in that: A content encoder is used to extract content features of a facial makeup image, and a makeup style encoder is used to extract the makeup style of the facial makeup image; the makeup style encoder uses a hierarchical convolutional neural network to capture makeup-related makeup styles in facial makeup images.

5. A multi-scale digital makeup migration method as claimed in claim 1, characterized in that: The face area is divided into several local face blocks. The makeup style of the face area is judged by a global discriminator with a multi-scale discriminant network structure. The facial makeup of the local face blocks is detected and corrected by a local discriminator. The effect of makeup transfer is evaluated based on the judgment and detection results of the global discriminator and the local discriminator.

6. A multi-scale digital makeup migration method as claimed in claim 1, characterized in that: The acquired facial makeup image is segmented into different semantic regions by using a generator that introduces a facial parsing graph. According to the obtained semantic regions, the background region of the facial makeup image is obtained, thereby completing the extraction of the background region. The facial parsing graph is used to extract the facial region pixels of the makeup migration image, thereby obtaining the facial region of the makeup migration image.

7. A multi-scale digital makeup migration system, characterized in that: include: An acquisition module, which is configured to acquire different facial makeup images; A migration module is configured to use a multi-scale digital network model to perform makeup migration of a facial makeup image to obtain a facial makeup image after makeup migration; specifically, in the process of performing makeup migration of a facial makeup image, extract content features, makeup styles and background areas of different acquired facial makeup images; perform makeup migration according to the content features of the extracted facial makeup image to obtain a makeup migration image, wherein the migrated makeup style is different from the makeup style of the facial makeup image to be migrated; extract a facial area of ​​the obtained makeup migration image; fuse the extracted background area and the facial area to obtain a facial makeup image after makeup migration, thereby realizing makeup migration of the facial makeup image; The multi-scale digital network model used adopts a generative adversarial network model based on a multi-discriminator architecture; the multi-discriminator architecture includes a content discriminator for facilitating the separation of content features and makeup styles, a global discriminator for evaluating the face area and identifying the quality and realism of makeup transfer of face makeup images, and a local discriminator for local makeup detection and correction of makeup transfer images; The content encoder is used to extract content feature representations from the input images without makeup and with makeup, which captures the content features in the image without considering the makeup style; The generator takes the original image and its corresponding face analysis map, as well as the facial features extracted by the content encoder and the makeup style encoder as input; Before inputting the makeup-free and makeup-removed images into the discriminator, the face parsing map is used to classify the image pixels; The specific steps of the local discriminator are: deforming and fusing the makeup image into the non-makeup image I s On the top, a synthetic makeup image W is generated m , using the face analysis graph to extract the synthetic makeup image W m , Makeup images And makeup image I m The facial area in the extracted synthetic makeup image is then extracted using the facial key points. Makeup image face area And the face area of ​​the makeup image Perform local block segmentation on I m Partial block of the face with makeup and The makeup face part P s Bfacei Input to the local discriminator corresponding to the local block In the example, let the discriminator judge the pair of local blocks with different makeup styles as negative, and I m Partial block of the face with makeup and W m Synthetic makeup face local block Input to In , let the discriminator judge the pair of local blocks with the same makeup style as positive.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the multi-scale digital makeup migration method according to any one of claims 1 to 6 are implemented.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the program, the steps of the multi-scale digital makeup migration method according to any one of claims 1 to 6 are implemented.

10. A computer program product comprising software code, characterized in that The program in the software code executes the steps of the multi-scale digital makeup migration method according to any one of claims 1 to 6.

Images