A semantic local discrimination method for facial makeup with reference conditions

Abstract

This invention discloses a semantic local discrimination method for facial makeup with reference conditions. It extracts semantic features from a reference image (with makeup) and the original image (without makeup). To ensure accurate makeup application in the synthesized image, a local discriminator with reference conditions is incorporated. Due to differences in facial structure and pose, a feature alignment network maps the features of the reference and original images to an aligned feature space and calculates feature statistics for regional makeup discrimination, ensuring accurate makeup transfer. Extensive experiments demonstrate the effectiveness and superior makeup transfer performance of this method across multiple makeup metrics, achieving state-of-the-art makeup transfer capabilities.

Description

Technical Field

[0001] This invention relates to the technical field of makeup transfer, and in particular to a method for semantic local discrimination of facial makeup with reference conditions. Background Technology

[0002] The goal of facial makeup transfer is to replicate the makeup style of a reference image onto a non-makeup face (the original image). The challenges of makeup transfer stem from unpaired training data and misalignment between the reference and original images. In recent years, to address these issues, existing methods have focused on spatially invariant makeup style transfer, synthesizing makeup images by injecting style information and transformations into the latent space of a generative adversarial network.

[0003] On the one hand, most makeup transfer methods capture facial makeup through style encoding, which has poor spatial representation capabilities. Furthermore, existing work on style information fusion is scarce. In real-world scenarios, the semantic regions of the original and reference face images are often misaligned. In cases of severe misalignment, injecting reference style codes into the makeup rendering network may lead to undesirable results. On the other hand, existing models often ignore the domain differences between the reference and synthesized images when measuring makeup consistency. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings and deficiencies of the prior art and propose a semantic local discrimination method for facial makeup with reference conditions, which can accurately transfer makeup. The local discriminator of this method has reference conditions, and effective feature fusion and statistical representation lead to makeup perception discrimination. Therefore, this method can achieve state-of-the-art makeup transfer capabilities.

[0005] To achieve the above objectives, the technical solution provided by this invention is: a method for semantic local discrimination of facial makeup with reference conditions, comprising the following steps:

[0006] S1: Prepare a face image dataset and a corresponding face segmentation mask dataset, which includes a reference image with elaborate makeup and an original image without makeup. Let the original image x and the reference image y be denoted as x∈X and y∈Y, respectively, where X is the original image dataset without makeup and Y is the reference image dataset with elaborate makeup.

[0007] S2: Prepare a generator implemented using a neural network. The generator produces a high-fidelity face image with a resolution of h×w using the original image x and the reference image y. Where h is the height of the image and w is the width of the image;

[0008] S3: Process the face image generated in step S2. The reference image y is fed into the global discriminator D. global The corresponding authenticity judgment result is obtained for the face image generated in step S2. The face segmentation mask of the reference image y is divided into regions, and the location and number of regions can be selected as needed.

[0009] S4: Convert the face image generated in step S2 The reference image y is multiplied by the corresponding face segmentation mask obtained in step S3 to obtain the local regions of the generated image for each segmented region. and the local region y of the reference image i Where i is the corresponding local region label, the resulting local region of the generated image Compared with the local region y of the reference image i Together they are fed into the local discriminator D local The true or false result is obtained by the determination;

[0010] S5: Through generator G and global discriminator D global Local discriminator D local The three elements constrain the learning of the neural network through the adversarial learning of the true and false judgment results. When the adversarial learning of the three elements reaches a balance, the generator can generate a face that accurately transfers the exquisite makeup of the reference image to the face of the original image.

[0011] Furthermore, in step S1, the prepared face image dataset is divided into LADN dataset, MT dataset and Makeup-Wild dataset according to the type of makeup. Each dataset includes face images without makeup as original images and face images with exquisite makeup as reference images. In the prepared face segmentation mask dataset, each face segmentation mask image corresponds to the corresponding face image. Each face segmentation mask image is segmented according to the face feature regions, and is divided into 14 regions: left eye region, right eye region, nose region, upper lip region, lower lip region, teeth region, left eyebrow region, right eyebrow region, left ear region, right ear region, face region, hair region, neck region and background region.

[0012] Further, in step S2, the generator requires a reference image and the original image as input. The generator first uses a multi-layer convolutional feature extractor to extract style features from the reference and original images. Then, it inputs the corresponding face segmentation mask and obtains 14 corresponding face image style features according to the regions divided by the face segmentation mask. Next, based on the makeup in the reference image to be transferred to the corresponding region in the original image, the corresponding style feature regions of the reference image are fused with the remaining style feature regions of the original image that do not require makeup transfer to obtain the fused style features. Then, the original and reference images are respectively fed into corresponding convolutional networks to extract the corresponding face image identity features. Then, the identity features of the reference and original images, the corresponding face segmentation mask, and the fused style features are fed together into a fused convolutional network for feature fusion. The output result is then fed into a convolutional network with the same structure for feature fusion. After repeatedly performing feature fusion through three convolutional networks with the same structure, the result is activated using the tanh activation function to obtain the final generated image. The generated image is represented by the following formula:

[0013]

[0014] In this formula, the face segmentation mask corresponding to each image is sent along with the image by default and is not shown.

[0015] Furthermore, in step S3, the global discriminator needs to input the corresponding image as a condition, and the discrimination loss function is as follows:

[0016] L adv =E y [log(D global (y))]+E x,y [log(1-D global (G(x,y)))]

[0017] L rec =E x,y [||G(G de (y,x),y)-y||1]

[0018] In the formula, L adv It is the resistance loss value; L rec It is the reconstruction loss value; E y E represents the expectation given the input y; x,y G represents the expected value given the inputs x and y; de () represents the makeup removal process, G() represents the makeup application process, and D() represents the makeup removal process. global () indicates the global discrimination process;

[0019] When dividing a region, some important parts on the face segmentation mask can be used as reference positions. The local region can be located by the distance between the required local region and the reference position, thereby realizing the division of the region.

[0020] Furthermore, the specific operation steps of step S4 are as follows:

[0021] S41: The obtained local regions of the generated image and the local regions of the reference image are represented by the following formula:

[0022]

[0023]

[0024] In the formula, ⊙ represents the dot product. This represents the face segmentation mask for the i-th local region of the original image. This represents the face segmentation mask for the i-th local region of the reference image; y represents a local region of the generated image, specifically the i-th local region image of the generated image; i This represents a local region of the reference image, specifically the i-th local region image of the reference image.

[0025] S42: After being fed into the local discriminator, it first goes through a feature alignment network for alignment, and its corresponding loss function is:

[0026]

[0027] In the formula, L layer This represents the loss value obtained from the feature alignment network. Representative Feature Alignment Network A fea Features extracted from the t-th convolutional layer Represents expectations, This represents the features obtained from the generated image. Represents features obtained from a reference image;

[0028] S43: Combine the feature and style encoder E obtained in step S42 sty For a local region y of the reference image i The extracted styles are fed into the feature style fusion module for fusion, expressed by the following formula:

[0029]

[0030]

[0031] In the formula, A fea () represents a feature alignment network. The features represent the fusion of generated image features and reference image style. The features represent the fusion of the reference image features and the reference image style;

[0032] S44: Will Each feature passes through its own convolutional layer, generating two branches. One branch obtains the corresponding features, which are then passed to the next layer as input. The other branch is used to calculate the corresponding static feature statistics, including the mean, maximum, and variance. The loss function for comparing the static feature statistics is as follows:

[0033]

[0034] In the formula, L reg This indicates that the loss value is calculated using static feature statistics. S represents the corresponding convolutional layer. k This represents the corresponding method for calculating static feature statistics; calculating static feature statistics allows for more precise makeup transfer.

[0035] S45: Flatten the obtained reference image features and generated image features into vectors, and input them into the fully connected layer for discrimination. The corresponding loss function is as follows:

[0036]

[0037]

[0038] In the formula, L sta This represents the loss value obtained after passing through the fully connected layer, and Φ represents a learnable linear mapping function to predict the probability that the region composition is true. For addition operation; l k () indicates the method for obtaining the discrimination result of the fully connected layer; i represents the i-th local region divided, and k represents the k-th static feature statistic;

[0039] S46: The vectors obtained in step S45 are used through contrastive learning to narrow the distance between real data and to widen the distance between real data and fake data. The loss function corresponding to contrastive learning is:

[0040]

[0041]

[0042] In the formula, L sty The contrarian learning loss value is represented by Ψ(), which represents a similarity calculation, and T() represents an affine transformation used for data augmentation; where, Positive samples; This is a negative sample.

[0043] Furthermore, in step S5, to ensure that the generated face image contains accurate semantic information of the face region and has the makeup of the reference image, while maintaining high realism and diversity, it is necessary to jointly train the generator, global discriminator, and local discriminator. These discriminators compete against each other, with the generator and local discriminator working together to guide the generator to capture accurate semantic information of the local region. During adversarial training, the loss functions of the generator, global discriminator, and local discriminator are defined as follows:

[0044]

[0045]

[0046] In the formula, E represents the feature information used in the generator to extract the original image and the reference image, and λ and ρ are weighting factors that control the relative importance of the corresponding regularization.

[0047] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0048] 1. Unlike existing models based on unconditional real synthetic makeup discrimination, this invention proposes a semantic local discrimination method for facial makeup with reference conditions from a new perspective, so as to promote more accurate makeup transfer.

[0049] 2. In order to guide the generator to pay more attention to makeup details, this invention improves the discrimination system architecture, performs reference condition feature extraction and fusion at the region level, and performs discrimination between real makeup and generated makeup based on static feature statistics.

[0050] 3. This invention designs a feature alignment network that aligns the features of the generated face image and the reference image at the region level, and further maps the features to the aligned feature space, which can effectively reduce the misalignment between the generated face image and the reference image.

[0051] 4. The generator and local discriminator of this invention share a style encoder to provide a reference image style. In the generator, it can be used to normalize the features of the reference image and generate face images to achieve effective feature fusion. In the local discriminator, it can provide an accurate reference image style for the discrimination process, enabling the local discriminator to better distinguish between real and fake data.

[0052] 5. The semantic local discrimination method for facial makeup with reference conditions proposed in this invention can more freely divide local areas according to its own needs, thus achieving superior performance in blush, multi-color eyeshadow, and complex makeup. Attached Figure Description

[0053] Figure 1This is a flowchart of the method of the present invention.

[0054] Figure 2 This is a structural diagram of the local discriminator in the method of the present invention. Detailed Implementation

[0055] The present invention will be further described in detail below with reference to the embodiments and accompanying drawings, but the embodiments of the present invention are not limited thereto.

[0056] like Figure 1 As shown, this embodiment discloses a method for semantic local discrimination of facial makeup with reference conditions, including the following steps:

[0057] S1: Prepare a face image dataset and a corresponding face segmentation mask dataset, which includes a reference image with elaborate makeup and an original image without makeup. Let the original image x and the reference image y be denoted as x∈X and y∈Y, respectively, where X is the original image dataset without makeup and Y is the reference image dataset with elaborate makeup.

[0058] The prepared face image datasets are divided into three datasets according to the type of makeup: LADN, MT, and Makeup-Wild. The MT dataset contains 1115 images without makeup and 2719 images with makeup, the Makeup-Wild dataset contains 369 images without makeup and 403 images with makeup, and the LADN dataset contains 333 images without makeup and 302 images with makeup. In the prepared face segmentation mask dataset, each face segmentation mask image corresponds to a face image. Each face segmentation mask image is segmented according to facial feature regions, into 14 regions: left eye region, right eye region, nose region, upper lip region, lower lip region, teeth region, left eyebrow region, right eyebrow region, left ear region, right ear region, face region, hair region, neck region, and background region.

[0059] S2: Prepare a generator implemented using a neural network. The generator produces a high-fidelity face image with a resolution of h×w using the original image x and the reference image y. Where h is the height of the image and w is the width of the image;

[0060] The generator requires a reference image and the original image as input. First, it uses a multi-layer convolutional feature extractor to extract style features from both the reference and original images. Then, it inputs a face segmentation mask and divides the image features into 14 regions based on the mask. Next, based on the makeup in the reference image to be transferred to the corresponding region in the original image, it fuses the relevant style feature regions of the reference image with the remaining style feature regions of the original image that do not require makeup transfer, resulting in a fused style feature. Then, the original and reference images are fed into their respective convolutional networks to extract corresponding face identity features. Finally, the identity features of the reference and original images, the corresponding face segmentation mask, and the fused style features are fed into a fused convolutional network for feature fusion. The output is then fed into a structurally identical convolutional network for further feature fusion. This process is repeated three times through three structurally identical convolutional networks. Finally, the result is activated using the tanh activation function to obtain the final generated image, which is represented by the following formula:

[0061]

[0062] In this formula, the face segmentation mask corresponding to each image is sent along with the image by default and is not shown.

[0063] S3: Process the face image generated in step S2. The reference image y is fed into the global discriminator D. global The corresponding authenticity judgment result is obtained for the face image generated in step S2. The face segmentation mask of the reference image y is divided into regions, and the location and number of regions can be selected as needed.

[0064] The global discriminator requires the corresponding image as input, and the loss function for discrimination is as follows:

[0065] L adv =E y [log(D global (y))]+E x,y [log(1-D global (G(x,y)))]

[0066] L rec =E x,y [||G(G de (y,x),y)-y||1]

[0067] In the formula, L adv It is the resistance loss value; L rec It is the reconstruction loss value; E y E represents the expectation given the input y;x,y G represents the expected value given the inputs x and y; de () represents the makeup removal process, G() represents the makeup application process, and D() represents the makeup removal process. global () indicates the global discrimination process;

[0068] When dividing a region, some important parts on the face segmentation mask can be used as reference positions. The local region can be located by the distance between the required local region and the reference position, thereby realizing the division of the region.

[0069] S4: Convert the face image generated in step S2 The reference image y is multiplied by the corresponding face segmentation mask obtained in step S3 to obtain the local regions of the generated image for each segmented region. and the local region y of the reference image i Where i is the corresponding local region label, the resulting local region of the generated image Compared with the local region y of the reference image i Together they are fed into the local discriminator D local The true / false result is obtained by determining the authenticity; the specific operation steps are as follows:

[0070] S41: The obtained local regions of the generated image and the local regions of the reference image are represented by the following formula:

[0071]

[0072]

[0073] In the formula, ⊙ represents the dot product. This represents the face segmentation mask for the i-th local region of the original image. This represents the face segmentation mask for the i-th local region of the reference image; y represents a local region of the generated image, specifically the i-th local region image of the generated image; i This represents a local region of the reference image, specifically the i-th local region image of the reference image.

[0074] S42: After being fed into the local discriminator, such as Figure 2 As shown, the alignment is first performed using a feature alignment network, and its corresponding loss function is:

[0075]

[0076] In the formula, L layer This represents the loss value obtained from the feature alignment network. Representative Feature Alignment Network A fea Features extracted from the t-th convolutional layer Represents expectations, This represents the features obtained from the generated image. Represents features obtained from a reference image;

[0077] S43: Combine the feature and style encoder E obtained in step S42 sty For a local region y of the reference image i The extracted styles are fed into the feature style fusion module for fusion, expressed by the following formula:

[0078]

[0079]

[0080] In the formula, A fea () represents a feature alignment network. The features represent the fusion of generated image features and reference image style. The features represent the fusion of the reference image features and the reference image style;

[0081] S44: Will Each feature passes through its own convolutional layer, generating two branches. One branch obtains the corresponding features, which are then passed to the next layer as input. The other branch is used to calculate the corresponding static feature statistics, including the mean, maximum, and variance. The loss function for comparing the static feature statistics is as follows:

[0082]

[0083] In the formula, L reg This indicates that the loss value is calculated using static feature statistics. S represents the corresponding convolutional layer. k This represents the corresponding method for calculating static feature statistics; calculating static feature statistics allows for more precise makeup transfer.

[0084] S45: Flatten the obtained reference image features and generated image features into vectors, and input them into the fully connected layer for discrimination. The corresponding loss function is as follows:

[0085]

[0086]

[0087] In the formula, L sta This represents the loss value obtained after passing through the fully connected layer, and Φ represents a learnable linear mapping function to predict the probability that the region composition is true. For addition operation; l k () indicates the method for obtaining the discrimination result of the fully connected layer; i represents the i-th local region divided, and k represents the k-th static feature statistic;

[0088] S46: The vectors obtained in step S45 are used through contrastive learning to narrow the distance between real data and to widen the distance between real data and fake data. The loss function corresponding to contrastive learning is:

[0089]

[0090]

[0091] In the formula, L sty The contrarian learning loss value is represented by Ψ(), which represents a similarity calculation, and T() represents an affine transformation used for data augmentation. Positive samples; This is a negative sample.

[0092] S5: Through generator G and global discriminator D global Local discriminator D local The three elements constrain the learning of the neural network through the adversarial learning of the true and false judgment results. When the adversarial learning of the three elements reaches a balance, the generator can generate a face that accurately transfers the exquisite makeup of the reference image to the face of the original image.

[0093] To ensure that the generated face images contain accurate semantic information about facial regions and retain the makeup of the reference image while maintaining high realism and diversity, a generator, a global discriminator, and a local discriminator need to be trained jointly. These discriminators compete against each other, with the generator and the local discriminator working together to guide the generator in capturing accurate semantic information about local regions. The loss functions for the generator, global discriminator, and local discriminator during adversarial training are defined as follows:

[0094]

[0095]

[0096] In the formula, E represents the feature information used in the generator to extract the original image and the reference image, and λ and ρ are weighting factors that control the relative importance of the corresponding regularization.

[0097] After training, the performance of this method was quantitatively evaluated on the LADN, MT, and Makeup-Wild datasets. The evaluation metrics included FID, LPIPS, PSNR, and SSIM. The experimental results are shown in Tables 1, 2, and 3.

[0098] Table 1. LPIPS loss for the LADN dataset

[0099]

[0100]

[0101] Table 2 shows the LPIPS loss for the MT dataset.

[0102]

[0103] Table 3 SSIM and PSNR metrics for LADN and MT datasets

[0104]

[0105] FID represents the similarity between the generated image and the real image in terms of feature distribution; the lower the value, the more realistic the generated image. LPIPS measures the difference between two images. It learns the inverse mapping from the generated image to the ground truth, forcing the generator to learn the inverse mapping from the fake image to reconstruct the real image, and prioritizes perceptual similarity between them; the lower the value, the more similar the two images. PSNR represents the ratio between the maximum possible power of the signal and the power of the destructive noise that affects the fidelity of its representation; the higher the value, the higher the reconstruction quality. SSIM is a metric that quantifies the structural similarity between two images; the higher the value, the more similar the two images. After evaluation, in terms of the FID index, this invention achieved 34.97, while the second lowest reached only 46.98, making this invention 12.01 lower than the second-ranked one; in terms of the LPIPS index, this invention achieved 0.1141±0.0453 for eyeshadow, while the second lowest reached only 0.1192±0.0513; in terms of lips, it achieved 0.1250±0.0492, while the second lowest reached only 0.1301±0.0510; in terms of skin tone, it achieved 0.2676±0.1062, while the second lowest reached only 0.2717±0.1110; in terms of the SSIM index, this invention achieved 0.9420, while the second highest was only 0.9379; in terms of the PSNR index, this invention achieved 21.4395, while the second highest was only 21.2181.

[0106] In addition, the present invention also divided the experimental results into ordinary makeup, complex makeup, and makeup with concealing effects, and conducted a user survey, the results of which are shown in Table 4.

[0107] Table 4: Selection Rate of Optimal Makeup Looks in User Survey

[0108] Data types Regular makeup Complex makeup Concealing makeup CycleGAN 4％ 4.875％ 5％ BeautyGAN 1.6％ 3.125％ 5.4％ PSGAN 1.2％ 1.75％ 1.8％ LADN 2.8％ 2％ 3.4％ SCGAN 4％ 14.375％ 0.4％ EleGANt 15％ 2.375％ 12.6％ SSAT 5％ 1％ 5％ This invention 63.4％ 70.5％ 66.4％

[0109] User research revealed that, under the same input image conditions, the optimal selection rate of this invention is at least 56% higher than the second-best result. The migration effect aligns with popular aesthetics and is worthy of promotion.

[0110] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A method for semantic local discrimination of facial makeup with reference conditions, characterized in that, Includes the following steps: S1: Prepare a face image dataset and a corresponding face segmentation mask dataset, which includes a reference image y with elaborate makeup and an original image x without makeup. Let the original image x and the reference image y be denoted as x∈X, y∈Y, where X is the original image dataset without makeup and Y is the reference image dataset with elaborate makeup. S2: Prepare a generator implemented using a neural network. Generate high-fidelity face images with a resolution of h×w Where h is the height of the image and w is the width of the image; S3: Process the face image generated in step S2. The reference image y is fed into the global discriminator D. global The corresponding authenticity judgment result is obtained for the face image generated in step S2. The face segmentation mask of the reference image y is divided into regions, and the location and number of regions are selected as needed; S4: Convert the face image generated in step S2 The reference image y is multiplied by the corresponding face segmentation mask obtained in step S3 to obtain the local regions of the generated image for each segmented region. and the local region y of the reference image i Where i is the corresponding local region label, the resulting local region of the generated image Compared with the local region y of the reference image i Together they are fed into the local discriminator D local The true / false result is obtained by determining the authenticity; the specific operation steps are as follows: After being fed into the local discriminator, the data is aligned using a feature alignment network with the following loss function: In the formula, L layer This represents the loss value obtained by the feature alignment network. Representative Feature Alignment Network A fea Features extracted from the t-th convolutional layer Represents expectations, This represents the features obtained from the generated image. Represents features obtained from a reference image; The obtained feature and style encoder E sty For a local region y of the reference image i The extracted styles are fed into the feature style fusion module for fusion, as shown below: In the formula, A fea () represents a feature alignment network. The features represent the fusion of generated image features and reference image style. The features represent the fusion of the reference image features and the reference image style; Will Each feature passes through its own convolutional layer, generating two branches. One branch obtains the corresponding features, which are then passed to the next layer as input. The other branch is used to calculate the corresponding static feature statistics, including the mean, maximum, and variance. The loss function for comparing the static feature statistics is as follows: In the formula, L reg This indicates that the loss value is calculated using static feature statistics. S represents the corresponding convolutional layer. k This represents the method for calculating the corresponding static feature statistics, where k represents the k-th static feature statistic. The obtained reference image features and generated image features are flattened into vectors and input into a fully connected layer for discrimination. The loss value L after passing through the fully connected layer is calculated. sta ; The resulting vectors are used to narrow the gap between real and fake data through contrastive learning, and to widen the gap between real and fake data. The contrastive learning loss value L is calculated. sty ; S5: Through generator G and global discriminator D global Local discriminator D local The three elements constrain the learning of the neural network through the adversarial learning of the true and false judgment results. When the adversarial learning of the three elements reaches a balance, the generator can generate a face that accurately transfers the exquisite makeup of the reference image to the face of the original image.

2. The method for semantic local discrimination of facial makeup with reference conditions according to claim 1, characterized in that, In step S1, the prepared face image dataset is divided into LADN dataset, MT dataset and Makeup-Wild dataset according to the type of makeup. Each dataset includes face images without makeup as original images and face images with exquisite makeup as reference images. In the prepared face segmentation mask dataset, each face segmentation mask image corresponds to the corresponding face image. Each face segmentation mask image is segmented according to the face feature regions, and is divided into 14 regions: left eye region, right eye region, nose region, upper lip region, lower lip region, teeth region, left eyebrow region, right eyebrow region, left ear region, right ear region, face region, hair region, neck region and background region.

3. The semantic local discrimination method for facial makeup with reference conditions according to claim 2, characterized in that, In step S2, the generator needs to input a reference image and the original image as conditions. The generator first uses a feature extractor with multiple convolutional layers to extract the style features of the reference image and the original image. Then, it inputs the corresponding face segmentation mask and obtains the face image style features of 14 regions according to the regions divided by the face segmentation mask. Then, according to the makeup of the reference image to be transferred to the corresponding region on the original image, the corresponding style feature region of the reference image is fused with the other style feature regions of the original image that do not need to be transferred to obtain the fused style features. Then, the original image and the reference image are fed into the corresponding convolutional networks to extract the corresponding facial image identity features; The identity features of the reference image and the original image, the corresponding face segmentation mask, and the fused style features are then fed into a fused convolutional network for feature fusion. The output is then fed into a structurally identical convolutional network for feature fusion. This process is repeated three times through three structurally identical convolutional networks. Finally, the result is activated using the tanh activation function to obtain the final generated image, which is represented by the following formula: The face segmentation mask for each image is sent along with the image by default.

4. The semantic local discrimination method for facial makeup with reference conditions according to claim 3, characterized in that, In step S3, the global discriminator needs to input the corresponding image as a condition, and the loss function for discrimination is as follows: L adv =E y [log(D global (y))]+E x,y [log(1-D global (G(x,y)))] L rec =E x,y [||G(G de (y,x),y)-y||1] In the formula, L adv It is the resistance loss value; L rec It is the reconstruction loss value; E y E represents the expectation given the input y; x,y G represents the expected value given the inputs x and y; de () represents the makeup removal process, G() represents the makeup application process, and D() represents the makeup removal process. global () indicates the global discrimination process; When dividing a region, some important parts on the face segmentation mask can be used as reference positions. The local region can be located by the distance between the required local region and the reference position, thereby realizing the division of the region.

5. The semantic local discrimination method for facial makeup with reference conditions according to claim 4, characterized in that, In step S4, the obtained local regions of the generated image and the local regions of the reference image are represented by the following formula: In the formula, ⊙ represents the dot product. This represents the face segmentation mask for the i-th local region of the original image. This represents the face segmentation mask for the i-th local region of the reference image; y represents a local region of the generated image, specifically the i-th local region image of the generated image; i This represents a local region of the reference image, specifically the i-th local region image of the reference image. Loss value L sta The corresponding loss function is as follows: In the formula, L sta This represents the loss value obtained after passing through the fully connected layer, and Φ represents a learnable linear mapping function to predict the probability that the region composition is true. For addition operation; l k () indicates the method for obtaining the discrimination result of the fully connected layer; i represents the i-th local region that has been divided. Contrast learning loss value L sty The corresponding loss function is: In the formula, L sty The contrarian learning loss value is represented by Ψ(), which represents a similarity calculation, and T() represents an affine transformation used for data augmentation; where, As a positive sample, This is a negative sample.

6. The semantic local discrimination method for facial makeup with reference conditions according to claim 5, characterized in that, In step S5, to ensure that the generated face image contains accurate semantic information of the face region and has the makeup of the reference image, while maintaining high realism and diversity, it is necessary to jointly train the generator, global discriminator, and local discriminator. These discriminators compete against each other, with the generator and local discriminator working together to guide the generator in capturing accurate local semantic information. During adversarial training, the loss functions of the generator, global discriminator, and local discriminator are defined as follows: In the formula, E′ represents the feature information used in the generator to extract the original image and the reference image, and λ and ρ are weighting factors that control the relative importance of the corresponding regularization.

Images