A makeup transfer model training method based on cross-identity triplets

Abstract

The application discloses a makeup transfer model training method and a makeup transfer method based on cross-identity triplets, and relates to the technical field of computer vision and generative artificial intelligence. The model training method is divided into two stages of offline data construction and model training. In the offline data construction stage, a makeup prompt word dictionary is generated by a large language model; a homologous local rendering is performed on a nude face image cluster to generate a made-up image; a face mask is used to perform structure, background and rendering effectiveness filtering and semantic consistency verification, and a large number of high-quality cross-identity triplets are constructed through cross combination; a pre-trained diffusion model is fine-tuned under the strong supervision of a fixed text prompt and a double-path image condition, a conversion from text control to pure image control is realized, and a makeup transfer model is obtained. The application solves the problems of cross-identity makeup data scarcity, text semantic ambiguity and dirty data interference, realizes precise, lossless and what-you-see-is-what-you-get makeup transfer, and is suitable for film and television makeup, digital person rendering and e-commerce virtual makeup testing scenes.

Description

Technical Field

[0001] This invention relates to the fields of computer vision, natural language processing and generative artificial intelligence, and in particular to a makeup transfer model training method and makeup transfer method based on cross-identity triples, which can be applied to high-quality AIGC content creation, film and television-level virtual makeup, high-fidelity rendering of digital human assets and e-commerce virtual makeup try-on scenarios. Background Technology

[0002] With the rapid development of generative artificial intelligence (AIGC) and diffusion modeling techniques, image editing and feature transfer technologies have made significant progress. In the fields of computational aesthetics and portrait processing, the demand for high-quality, high-precision makeup transfer is becoming increasingly urgent. Currently, generative makeup editing mainly falls into two major technical routes: text-driven editing and reference image-driven transfer.

[0003] In high-end commercial applications such as film and television post-production and virtual makeup try-ons for brands, users demand a precise "what you see is what you get" effect, that is, to accurately replicate the makeup onto a source image of bare skin based on a reference makeup image. However, existing image-driven large-scale models face insurmountable data barriers and semantic gaps when trying to achieve this goal, with the following specific shortcomings: First, high-quality cross-identity triplet data is difficult to obtain, posing a physical challenge in data collection. Training a generative model capable of accurately replicating makeup requires massive amounts of high-quality triplet data: images of bare faces, reference images with makeup, and real-life target images with makeup. In reality, it's nearly impossible to find two people with different appearances but identical makeup details; manual drawing or retouching is extremely costly, making it difficult to meet the large-scale data requirements for training large models. Using augmentation with data of the same identity as a substitute can easily lead to the model directly copying the identity of the reference image during transfer, failing to truly learn cross-identity makeup transfer capabilities.

[0004] Second, text-driven models suffer from semantic ambiguity, making it difficult to guarantee makeup consistency. Existing technologies often use text-based prompts to generate cross-identity data, but natural language descriptions are abstract and vague. When the same makeup prompt is applied to different facial structures, it can generate makeup with vastly different color distributions, eyeliner directions, and detail textures. This makes it impossible to form strictly defined cross-identity identical makeup triples, leading to feature confusion during downstream model training and unstable makeup transfer effects.

[0005] Third, the lack of a targeted multi-level data cleaning mechanism means that dirty data can easily damage model performance. Existing generative models are prone to problems such as structural illusion, facial distortion, and background manipulation when synthesizing data. Current technologies lack multi-dimensional deep cleaning and human-machine collaborative filtering mechanisms tailored to the characteristics of makeup generation. Directly using dirty data containing distortion and noise for model fine-tuning will severely damage the original facial identity preservation ability of the pre-trained model, resulting in poor makeup transfer effects, low identity consistency, and background distortion.

[0006] In summary, existing technologies cannot construct cross-identity makeup triplet data at low cost and with high quality, nor can they achieve the effect of accurately transferring makeup without relying on text or only on reference images, which restricts the application of makeup transfer technology in high-end commercial scenarios. Summary of the Invention

[0007] To address the aforementioned shortcomings of existing technologies, this invention provides a method for training a makeup transfer model based on cross-identity triples and a makeup transfer method.

[0008] One aspect of the present invention provides a method for training a makeup transfer model based on cross-identity triples, including an offline data construction stage and a model training stage; The offline data construction phase includes: S1. Construct a priori cue word dictionary: Utilize a pre-trained large language model to generate fine-grained makeup cue words covering multi-dimensional makeup attributes, forming a cue word dictionary. ; S2, Local Rendering of Same-Source Clusters: Acquire images of unmade-up faces and divide them into multiple image clusters. Each image cluster contains... Images of Zhang from different backgrounds without makeup For each image cluster, select the same makeup prompt word from the prompt word dictionary. A local text image editing model is used to generate corresponding makeup image clusters while preserving the facial topology and non-facial background regions. ; S3. Multi-dimensional Mask Cleaning: Extract the face mask for each bare-faced image and its corresponding made-up image. Perform structural alignment filtering, background consistency filtering, and rendering validity filtering based on the face mask. Then, remove samples with inconsistent makeup through semantic consistency verification to obtain... Zhang is a valid sample; S4. Construction of cross-identity triples: [The sentence is incomplete and requires further context.] Zhang's valid samples are arranged and combined to generate cross-identity triples, which include a bare-faced source image, a makeup reference image, and a corresponding makeup target image. The model training phase includes: S5. Based on the pre-trained text conditional diffusion model, the input text prompt words are fixed as the unified identifier; the bare-faced source image and the makeup reference image in the cross-identity triple are used as dual-path image conditional inputs, and the makeup target image is used as the supervision signal for supervised training to obtain the image conditional diffusion model.

[0009] Another aspect of the present invention provides a makeup transfer method, the makeup transfer method comprising: S6. Image Input and Preprocessing: Receive the user's input of a raw image without makeup and a reference image with makeup, and perform normalization preprocessing. S7. Image Conditional Makeup Transfer: Input the preprocessed bare-faced source image, the makeup reference image, and the unified identifier into the trained image conditional diffusion model. The model outputs the target image after makeup transfer. The trained image conditional diffusion model is trained according to the aforementioned training method.

[0010] The present invention has the following beneficial effects: 1. Overcoming the bottleneck of cross-identity data acquisition, constructing high-quality triplet datasets at low cost. This invention uses a large language model to generate diverse makeup prompts and employs a homogeneous cluster rendering and cross-combination strategy to generate massive, pixel-level corresponding cross-identity makeup triplet data in batches without relying on real-world scene collection or manual refinement. This solves the technical dilemma of obtaining high-quality cross-identity makeup in the physical world, significantly reduces data preparation costs, and meets the data requirements for large-scale training of generative models.

[0011] 2. Multi-level cleaning and consistency verification effectively avoid interference from dirty data and ensure model training quality. This invention constructs a dual purification mechanism of automated mask filtering and human-machine collaborative verification. Through triple screening of structural alignment, background consistency, and rendering effectiveness, it removes unqualified samples such as facial distortion, background tampering, and makeup failure; and through semantic consistency verification, it eliminates visual ambiguity of makeup caused by text-driven processing, ensuring that the training data is pure and accurately labeled, avoiding the negative impact of dirty data on model fine-tuning, and effectively preserving the original facial identity consistency and generation clarity of the pre-trained model.

[0012] 3. Eliminating semantic ambiguity in text and achieving high-precision makeup transfer driven by pure images. This invention eliminates the model's dependence on text descriptions by fixing text prompts and introducing a fine-tuning strategy with dual-path image conditional inputs. This forces the model to directly extract high-frequency makeup features from the reference image and accurately transfer them to the target face, overcoming the makeup inconsistency problem caused by semantic ambiguity in text-driven models. It achieves true "what you see is what you get" cross-identity makeup replication, significantly improving the accuracy and stability of makeup transfer.

[0013] 4. Leveraging the prior knowledge of pre-trained models lowers the barrier to technology implementation and adapts to various high-end application scenarios. This invention eliminates the need to build a generative model from scratch. Instead, it performs paradigm transfer based on a mature pre-trained diffusion model, inheriting the powerful prior capabilities of the base model in understanding light and shadow, skin texture, and spatial structure. This significantly shortens the development cycle and reduces training costs. The finely tuned model can be directly deployed in industrial-grade scenarios such as film and television virtual makeup creation, high-fidelity rendering of digital humans, e-commerce virtual makeup try-on, and AIGC content creation, demonstrating significant industrial application value. Attached Figure Description

[0014] Figure 1 This is a flowchart of a makeup transfer model training method based on cross-identity triples according to a preferred embodiment of the present invention.

[0015] Figure 2 This is a flowchart of a makeup migration method according to a preferred embodiment of the present invention. Detailed Implementation

[0016] The present invention will be further illustrated below through embodiments, which are intended only to better illustrate the content of the present invention and not to limit the scope of protection of the present invention.

[0017] A preferred embodiment of the present invention provides a makeup transfer model training method based on cross-identity triples. The overall process includes an offline data construction stage and a model training stage, which are described below in conjunction with... Figure 1 The process of the model training method of the present invention will be described in detail.

[0018] The offline data construction phase includes steps S1 to S3. The first step, S1, is the construction of the prior cue word dictionary. In this embodiment, a pre-trained Large Language Model (LLM) is invoked. An instruction template containing multi-dimensional makeup attributes such as base makeup texture, eyeshadow color scheme, eyeliner style, lip texture, and overall makeup style is input. The LLM automatically generates fine-grained cue words covering everyday light makeup, heavy makeup, and stylistic makeup looks. Examples of cue words for lip makeup include: matte deep red lipstick, glossy cherry red lip gloss, satin dusty rose lipstick, and metallic copper lipstick; examples of cue words for eye makeup include: classic black eyeliner, soft brown smoky eyeshadow, metallic gold cut-out eyeshadow, and neon pink creative line eyeliner. These cue words refine makeup features from the dimensions of base makeup, eye makeup, lip makeup, contouring, and overall style. The number of cue words ranges from hundreds to thousands, and after aggregation, a structured cue word dictionary is formed. This provides rich and diverse semantic priors for makeup in subsequent data synthesis.

[0019] After constructing the prompt word dictionary, the S2 homogeneous cluster local rendering step is performed. A large number of real, unmade-up human face images are obtained from a public image database. Samples with clear faces, no occlusion, and clean backgrounds are selected. These unmade-up images are randomly divided into multiple image clusters, each containing... Images of people from different backgrounds without makeup, i.e., image clusters From the prompt word dictionary Select a single makeup prompt word The input is a local text image editing model with region feature preservation capabilities. During the rendering process, the model strictly constrains the facial topology, facial feature proportions, face shape contours, and non-facial background areas of each bare face image to remain unchanged. Only the facial area is rendered with the target makeup according to the prompt words, thereby generating a cluster of made-up images that corresponds one-to-one with the cluster of bare face images. This ensures that all images with makeup within the same image cluster follow the same makeup semantics.

[0020] The next step is the S3 multi-dimensional mask cleaning process. For the rendered clusters of images with makeup, a face segmentation model is first used to extract the face mask for each image without makeup and the corresponding image with makeup. The mask area includes skin, facial features, and glasses, used to accurately distinguish the face from the background. Then, automated filtering is performed, preferably sequentially including the following three dimensions: (1) Calculate the intersection-over-union (IoU) ratio of the facial masks before and after editing. Preferably, the IoU threshold is set to 0.9, and images with facial position shifts or deformations exceeding the set threshold are removed. Preferably, the intersection-over-union ratio of the facial masks is calculated using a face parsing algorithm. The facial mask intersection-over-union ratio (IoU) is the intersection area of ​​the unedited bare face mask and the edited made-up face mask divided by the union area. It is used to measure whether the face position, contour, and size have been significantly shifted or deformed.

[0021] (2) Calculate the mean square error (MSE) of the background region pixels. Preferably, the MSE threshold is set to 10 to remove unqualified images with background noise, distortion, or tampering. The mean square error (MSE) of the background region pixels refers to the average of the squares of the differences between the background pixel values ​​of the unedited bare image and the background pixel values ​​of the edited made-up image within the common background region before and after editing. It is used to determine whether the background has noise, distortion, or tampering. The smaller the MSE, the more likely the background has not changed, is clean, and has not been tampered with. MSE > 10 (threshold) indicates that the background has changed significantly (noise, overall offset, distortion, incorrect background region editing, etc.).

[0022] (3) Calculate the L1 distance of the facial region pixels. Preferably, the L1 threshold is set to 5 to remove images with unclear makeup rendering or that do not effectively respond to prompts. The L1 distance of the facial region pixels is calculated by subtracting the pixels of the bare face before editing and the pixels of the face with makeup after editing point by point within the face mask, taking the absolute value, and then averaging them to determine whether the makeup has been truly applied to the face.

[0023] After automated filtering, semantic consistency is verified by combining manual sampling comparison with algorithm feature similarity matching. Abnormal samples generated from the same prompt word but with significant differences in makeup color and style are removed, and finally, only the following samples are retained. The effective samples with highly consistent makeup and acceptable image quality correspond to bare-faced images. Images with makeup are .

[0024] S4 Cross-Identity Triple Construction Steps: Effective samples are cross-combined, and samples are randomly selected. (1≤ ≤ (unfaced image) As the source image, samples were selected. Images with makeup As a reference image, the sample Images with makeup That is, the target image, from which it is generated Cross-identity triplet Each triplet contains a source image with no makeup, a reference image with makeup, and a target image with makeup. Here, because... , The source image was generated from the same prompt word and passed the consistency check. Corresponding makeup photos This refers to high-quality cross-identity transfer tags. In other words, the cross-identity transfer tags are those associated with the original, unedited image. Same identity, and similar to the reference image with makeup. Target images with the same makeup look , used as the supervised ground truth for model training, characterizing the standard results of cross-identity makeup transfer.

[0025] After constructing the cross-identity triplet dataset, proceed to step S5 of the model training phase. Select a text-based conditional diffusion model as the base model, preferably a pre-trained DiT (DiT, Diffusion Transformer) model based on the Transformer-based diffusion flow matching architecture. During training, the text prompts originally received by the model are uniformly fixed to semantically neutral identifier strings, such as the fixed string "makeup," thereby blocking the model's dependence on text semantics. The plain-faced source images from the cross-identity triplet dataset are then used... Reference image with makeup Concatenated along the sequence dimension, serving as a dual-path image conditional input model; the target image with makeup... Add Gaussian noise Subsequently, as the latent variable to be denoised, the model predicts the noise in the latent variable and calculates its error with the real noise, then performs backpropagation to iteratively update the model parameters until the model loss converges, completing the transformation from text-controlled to image-controlled, resulting in an image conditional diffusion model capable of purely image-driven makeup transfer. Through this strongly supervised fine-tuning of the real triples, the large model completely changes its underlying dependencies, learning to directly learn from unfamiliar... Extract high-frequency color features and accurately map them to... Face.

[0026] Figure 2 This is a flowchart of a makeup transfer method according to a preferred embodiment of the present invention, including step S6 (image input and preprocessing) and step S7 (image conditional makeup transfer). First, in step S6 (image input and preprocessing), the user uploads a raw image of their bare face (the image to be transferred) through a terminal interface. and makeup reference images containing the target makeup look. The system performs uniform resolution cropping on the two input images, for example, cropping them to 512×512 pixels, and normalizes the pixel values ​​to the range of [-1,1], so that the image size and numerical range are adapted to the input requirements of the image conditional diffusion model.

[0027] After preprocessing, perform S6 image-driven makeup transfer and output the results. The processed raw image of the face is then used. Makeup reference images Together with a fixed, semantically unidentified string, the image is input into the trained image conditional diffusion model. The model performs stream matching sampling for denoising according to a preset number of steps, preferably 20 steps of stream matching sampling, gradually generating images from pure noise and accurately extracting the reference image from the latent space. High-frequency makeup features were analyzed and accurately transferred to the raw image of bare skin. Based on the facial topology, the final image of the target with makeup is obtained by reconstructing it using a VAE decoder. The output target image fully preserves the facial identity, facial features, and background content of the original image without makeup, while replicating the makeup colors, textures, and details of the reference image at the pixel level, achieving a high-precision, high-fidelity cross-identity makeup transfer effect.

[0028] Obviously, those skilled in the art should recognize that the above embodiments are only used to illustrate the present invention and are not intended to limit the present invention. Any changes or modifications to the above embodiments that are within the essential spirit of the present invention will fall within the scope of the claims of the present invention.

Claims

1. A method for training a makeup transfer model based on cross-identity triples, characterized in that, This includes the offline data construction phase and the model training phase; The offline data construction phase includes: S1. Construct a priori cue word dictionary: Utilize a pre-trained large language model to generate fine-grained makeup cue words covering multi-dimensional makeup attributes, forming a cue word dictionary. ; S2, Local Rendering of Same-Source Clusters: Acquire images of unmade-up faces and divide them into multiple image clusters. Each image cluster contains... Images of Zhang from different backgrounds without makeup For each image cluster, select the same makeup prompt word from the prompt word dictionary. A local text image editing model is used to generate corresponding makeup image clusters while preserving the facial topology and non-facial background regions. ; S3. Multi-dimensional Mask Cleaning: Extract the face mask for each bare-faced image and its corresponding made-up image. Perform structural alignment filtering, background consistency filtering, and rendering validity filtering based on the face mask. Then, remove samples with inconsistent makeup through semantic consistency verification to obtain... Zhang is a valid sample; S4. Construction of cross-identity triples: [The sentence is incomplete and requires further context.] Zhang's valid samples are arranged and combined to generate cross-identity triples, which include a bare-faced source image, a makeup reference image, and a corresponding makeup target image. The model training phase includes: S5. Based on the pre-trained text conditional diffusion model, the input text prompt words are fixed as the unified identifier; the bare-faced source image and the makeup reference image in the cross-identity triple are used as dual-path image conditional inputs, and the makeup target image is used as the supervision signal for supervised training to obtain the image conditional diffusion model.

2. The method according to claim 1, characterized in that, In step S1, the multi-dimensional makeup attributes include base makeup texture, eyeshadow color scheme, eyeliner style, lip texture, and makeup style; the prompt word dictionary contains hundreds to thousands of different makeup prompt words.

3. The method according to claim 1, characterized in that, In step S2, the local text image editing model has the ability to preserve regional features, and does not change the proportion of facial features, facial contours and background content during the rendering process.

4. The method according to claim 1, characterized in that, In step S3, the structure alignment filtering involves calculating the cross-union ratio (IoU) of the facial masks before and after editing, and discarding images whose facial position shifts or deformities exceed the IoU threshold. The background consistency filtering method calculates the mean square error (MSE) of the background region before and after editing, and removes images with altered backgrounds. The rendering validity filtering involves calculating the L1 pixel distance between the facial regions before and after editing, and discarding images that are not effectively made up.

5. The method according to claim 1, characterized in that, In step S3, the semantic consistency verification includes manual verification and algorithm similarity verification, which is used to remove abnormal samples with inconsistent colors or styles generated by the same prompt word.

6. The method according to claim 1, characterized in that, In step S4, the permutation and combination is: arbitrarily selecting samples from the valid samples. bare face image As the source image, samples were selected. Images with makeup As a reference image, the sample Images with makeup That is, the target image, from which it is generated Cross-identity triplet Each triplet contains a raw image of the subject. Makeup reference images and images of targets with makeup ,in .

7. The method according to claim 1, characterized in that, In step S5, the pre-trained text conditional diffusion model is a DiT model based on the Transformer diffusion flow matching architecture; the fixed input text prompt words are used to block the model's dependence on text semantics.

8. A makeup transfer method, characterized in that, The makeup transfer method includes: S6. Image Input and Preprocessing: Receive the user's input of a raw image without makeup and a reference image with makeup, and perform normalization preprocessing. S7. Image Conditional Makeup Transfer: Input the preprocessed bare-faced source image, the makeup reference image, and the unified identifier into the trained image conditional diffusion model, and output the target image after makeup transfer from the model. The trained image conditional diffusion model is trained by the training method according to any one of claims 1 to 7.

9. A makeup transfer method, characterized in that, In step S7, the image conditional diffusion model uses 20-step flow matching sampling to generate a denoised image. The output target image retains the identity and background of the source image and is consistent with the makeup of the reference image at the pixel level.

Images