Face make-up real-time generation method based on two-stage diffusion model

Abstract

The invention discloses a face makeup real-time generation method based on a two-stage diffusion model, and the method comprises the steps: carrying out the preprocessing of a face-free face image of a user, analyzing the color and contour features of the face of the user, obtaining the personal color information of different users, and classifying the personal color information into a predefined color type, namely, the personal color analysis results of different users; on the basis of a two-stage diffusion model, generating a face makeup image according to the face-free face image uploaded by the user and the personal color analysis result; and carrying out refinement post-processing and optimization post-processing on the generated face makeup image. According to the method, the makeup effect can be finer, accurate adjustment can be carried out according to the individual requirements of the user, it is ensured that the finally generated makeup image is more attractive visually, and meanwhile the stability of the generation model is improved.

Description

Technical Field

[0001] The present invention belongs to the field of image processing and generative artificial intelligence application technology, relates to image processing, and in particular to a real-time generation method of facial makeup based on a two-stage diffusion model. Background Art

[0002] Facial makeup generation is an important research direction in the field of computer vision and image processing. It not only has broad application prospects in the entertainment and beauty industries, but also plays a key role in virtual makeup trials, film and television production, and social media. Traditional makeup generation methods usually rely on image processing technology and template matching. Although they can achieve makeup effects to a certain extent, they often have problems such as lack of naturalness and poor flexibility. Summary of the invention

[0003] In order to overcome the above-mentioned shortcomings of the prior art, the object of the present invention is to provide a real-time facial makeup generation method based on a two-stage diffusion model to achieve better facial makeup generation effect.

[0004] In order to achieve the above object, the technical solution adopted by the present invention is:

[0005] A method for real-time generation of facial makeup based on a two-stage diffusion model comprises the following steps:

[0006] Step 1, preprocessing the user's face image without makeup to analyze the color and contour features of the user's face, obtain personal color information of different users, and classify them into predefined color types, that is, personal color analysis results of different users;

[0007] Step 2: Based on the two-stage diffusion model, a facial makeup image is generated according to the bare face image uploaded by the user and the personal color analysis result;

[0008] Step 3, performing refinement post-processing and optimization post-processing on the generated facial makeup image.

[0009] Compared with the prior art, the present invention has the following beneficial effects:

[0010] First, by deeply analyzing the color information of the user's iris, hair, and skin areas, we can ensure that the generated makeup is highly consistent with the user's personal color information, thereby improving the personalization of the generated effect. This process not only improves the diversity of makeup, but also makes the generated makeup more in line with the user's aesthetic expectations.

[0011] Secondly, compared with the traditional single-stage generation method, the two-stage diffusion model of the present invention has a significant improvement in makeup details and naturalness. By dividing the generation process into two stages, the detail generation of each step can be better controlled, making the makeup effect more delicate and able to be accurately adjusted according to the user's personalized needs, ensuring that the final generated makeup image is more visually attractive, while also improving the stability of the generation model.

[0012] Finally, combined with the user feedback mechanism, the present invention can continuously adjust and optimize the generation parameters according to the needs of the user, so that the makeup effect generated each time is more in line with the user's personalized needs, enhancing the user experience, thereby significantly improving the flexibility and adaptability of the present invention.

[0013] In addition, by integrating the model into the TensorRT engine, the present invention can significantly improve the speed of makeup generation. Users can see personalized makeup effects in a short time and realize real-time generation of makeup images, which has broad application value in scenarios such as virtual makeup trial, live broadcast and social media. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 It is a schematic diagram of the process of the present invention.

[0015] Figure 2 The facial makeup images are generated by using the dataset collected by the present invention to train the network.

[0016] Figure 3 This is the image effect generated by makeup on an existing foreign open source dataset using the method of the present invention.

[0017] In each picture, the eyes of the face area are blurred. DETAILED DESCRIPTION

[0018] The embodiments of the present invention are described in detail below with reference to the accompanying drawings and examples.

[0019] The existing makeup generation technology has not yet taken personalized factors into consideration, resulting in a lack of personalization in the generated effect, which is merely a mechanical transplantation of existing makeup.

[0020] To this end, the present invention adopts a real-time generation method of facial makeup based on a two-stage diffusion model. Figure 1 As shown, it mainly includes the following steps:

[0021] Step 1: pre-process the user's bare face image to analyze the color and contour features of the user's face, obtain personal color information of different users, and classify them into predefined color types, that is, personal color analysis results of different users.

[0022] This step is used to analyze the color and features of the user's face, and is mainly preprocessed through the following process:

[0023] Step 11, detect the user's plain face image to obtain the face area, and the face area of ​​the present invention is further subdivided into iris area, hair area and skin area. For example, detecting n user's plain face images can obtain n face areas, and correspondingly obtain n iris areas, n hair areas and n skin areas.

[0024] Specifically, for the bare-faced photos uploaded by users, the HOG features and linear classifier are used to detect the facial area in the image. Within the detected facial area, the pre-trained 68-point landmark detection model is used to extract facial key points to obtain the user's face area. Subsequently, the color of the user's face is analyzed, and the facial landmarks are used to determine the areas of the iris, hair, and skin.

[0025] Step 12, extracting average color values ​​from the iris area, hair area and skin area of ​​the face area respectively, and converting the extracted color information from the initial RGB color space to the Lab color space to obtain the Lab color space information of the above three areas.

[0026] For the i-th user, the average color values ​​are extracted from the iris area, hair area, and skin area, respectively, and are represented as i1, i1, and i3. They are converted into Lab color space information, respectively, and are represented as and Then, the Lab color space information of the i-th user is expressed as follows:

[0027] The conversion method of the present invention for converting RGB color into Lab color space is as follows:

[0028] First convert RGB to XYZ color space:

[0029] X=0.4124564R+0.3575761G+0.1804375B

[0030] Y=0.2126729R+0.7151522G+0.0721750B

[0031] Z=0.0193339R+0.1191920G+0.9503041B

[0032] Then convert XYZ to Lab color space:

[0033] L=116f(Y / Y n )-16

[0034] a=500(f(X / X n)-f(Y / Y n ))

[0035] b=200(f(Y / Y n )-f(Z / Z n ))

[0036] Where f(t) = t 1 / 3 If t>0.008856, otherwise f(t)=7.787t+16 / 116.

[0037] Step 13, cluster the Lab color space information of the iris area, hair area and skin area. Through cluster analysis, this information will be further converted into different color types, that is, the personal color analysis results of different users, such as predefined color types such as warm and cold tones or seasonal types. The purpose of this step is to provide a reliable personalized color foundation for subsequent makeup generation, ensuring that the final makeup can meet the personalized needs of users.

[0038] This step groups the extracted Lab color space information of the iris, hair, and skin areas through a clustering algorithm (such as K-means clustering). Specifically, the user's personal color information is first extracted, and then the eigenvalues ​​in the Lab color space are used to define the cluster centers. By calculating the distance between the user's personal color information and these cluster centers, the user's color information is classified into the closest color type, thereby achieving color type classification. This adjustment is based on the division of different areas in the color space. By optimizing the position of the cluster center, personal color information can be accurately classified into cool and warm tones or seasonal types.

[0039] Specifically, the cluster center is set in advance according to the warm and cold tones and season types, denoted as μ j , where j = 1, 2, ..., k. These centers can be initialized according to known color types, such as cold colors (representing winter and summer) and warm colors (representing spring and autumn). These centers are initialized using existing color datasets.

[0040] This step uses K-means clustering, with μ j is the cluster center. The goal of K-means is to minimize the Euclidean distance between the sample and the cluster center. The K-means objective function J is:

[0041]

[0042] Where k is the number of personal color types, r ij is an indicator variable, when the sample x i 1 if it belongs to the jth cluster, otherwise 0; ∥∥x i -μ j ∥∥2 Represents sample x i and cluster center μ j The Euclidean distance of , ∥∥x i -μ j ∥∥ 2 =(L i -L j ) 2 +(a i -a j ) 2 +(b i -b j ) 2 .

[0043] In this step, K-means clustering iteratively optimizes the cluster center μ j To achieve this, the specific steps are as follows:

[0044] 1) Allocation step: for each sample x i , find the nearest cluster center μ j ,Right now:

[0045]

[0046] 2) Update step: Recalculate the cluster center μ according to the distribution of samples j The location, that is:

[0047]

[0048] This formula is obtained by summing up all samples x belonging to the same cluster. i The average value is used to update the location of the cluster center.

[0049] Through the above iterative process, the center μ of each cluster will be obtained. j , and each user's Lab color information x i Assign to the nearest cluster center μ j These cluster centers are preset as warm and cold tones and seasonal types, and the user's color information is divided into one of these types. Through the K-means objective function J, the distance from the sample to the cluster center can be minimized. Thus, the user's personal color information is assigned to the category corresponding to "warm and cold tones and seasonal types".

[0050] Step 2: Based on the two-stage diffusion model, a facial makeup image is generated according to the bare face image uploaded by the user and the personal color analysis results.

[0051] The diffusion model is a type of generative model based on the probabilistic diffusion process. Its basic idea is to restore the image by gradually denoising it, thereby generating high-quality images. Compared with GAN, the diffusion model has certain advantages in the quality and diversity of generated images.

[0052] In this step, a two-stage diffusion model is designed in the makeup generation process. The uploaded user's bare face image is used as input, and the user's personal color analysis results are used as conditional information, which is embedded in the feature representation of the input image to guide the makeup generation process. Among them, the first stage is the basic generation of makeup. Based on the user's personalized color conditions, a preliminary makeup image is gradually constructed through the diffusion model. The second stage further refines and optimizes the makeup image to enhance the details and naturalness of the makeup. The purpose of this stage is to ensure that the generated makeup image is more realistic in visual effect and can meet the user's personalized aesthetic needs.

[0053] The two-stage diffusion model of the present invention generates a facial makeup image by a step-by-step denoising method. That is, in the forward diffusion process, the plain face image x0 is gradually converted into a pure noise image by adding noise. In the reverse diffusion process, the pure noise image is gradually converted into the target makeup image x0 by removing noise and combining the personal color analysis results. T , that is, facial makeup image.

[0054] Specifically, in the forward diffusion process (adding noise), the plain face image x0 is converted into the noisy image x t , the implementation method is as follows:

[0055]

[0056] in, is a parameter that controls the amount of noise. As t increases, the amount of noise increases. t is the time step. I is the identity matrix. q(x t |x0) represents the probability distribution of the data point at time t given the initial data point x0; is a Gaussian (normal) distribution, where x t At the data point at time t, the distribution is is the mean value, is the covariance matrix.

[0057] The present invention sets that when one of the following conditions is met, the forward diffusion process is terminated:

[0058] Condition 11: When t is equal to the preset fixed time step number T, the data has been covered by enough noise, and all data can evenly reach the state of high noise;

[0059] Condition 12, when the noise level reaches a set threshold, the threshold is sufficient to make any significant difference between the original data and the noise unrecognizable;

[0060] In the reverse diffusion process (denoising), without introducing the results of combining personal color analysis, the diffusion formula is as follows:

[0061] p θ (x t-1 ∣x t )=N(x t-1 ;μ θ (x t ,t),Σ θ (x t ,t))

[0062] Among them, μ θ and Σ θ are learnable parameters used in the denoising process, specifically:

[0063] p θ (x t-1 ∣x t ) means that at a given x t Generate x under the condition t-1 The probability distribution of . It is a conditional probability used in one step of the inverse diffusion process, that is, in the diffusion model from the noisy image x t Restore to the previous time step x t-1 In other words, it is the probability that the model learns to progressively denoise the noisy data to recover the final image.

[0064] μ θ (x t ,t) represents the learnable mean parameter, which is the conditional noise image x at time step t t This mean will guide how to gradually denoise the noisy image and approach the target makeup image x in the reverse diffusion process. T This means that after training the neural network, the model will learn the appropriate mean μ θ , so that noise can be accurately removed at each step.

[0065] Σ θ (x t ,t) represents the learnable covariance matrix, which describes the covariance matrix between the input x and the time step t. t The uncertainty (i.e., variance) of the noise distribution output by the model. The covariance matrix defines the noise size at a certain time step during the denoising process and controls the randomness and generation quality during the image restoration process.

[0066] When combined with the personal color analysis results, a facial makeup image can be generated according to the user's facial color conditions. At this time, the pure noise image is converted into the target makeup image x T , the implementation method is also the corresponding diffusion formula:

[0067] p θ (x t-1 ∣x t ,c)=N(x t-1 ;μ θ (x t ,t,c),Σ θ (x t ,t,c))

[0068] c represents the user's personal color analysis result, which serves as the personalized condition information for generating makeup and guides the model to generate makeup that meets personalized needs.

[0069] The present invention sets that when one of the following conditions is met, the reverse diffusion process is terminated:

[0070] Condition 21, starting from the time step when the forward diffusion process terminates, the data is gradually restored to time step 0. When the time step reaches 0, the output is regarded as the restored clean data;

[0071] Condition 22: When the makeup data quality (clarity and generation error) reaches a preset threshold. For example, the clarity is not less than the original Figure 3 %, and the error is less than 0.01.

[0072] In the past, the diffusion model input was only an image, but in the present invention, the user's color analysis results are input simultaneously. First, the user's iris, hair, and skin areas are preprocessed to extract personal color information. Through color extraction and Lab color space conversion, the color information is further clustered (such as warm and cold tones and season types). The main goal of this step is to provide personalized color condition information for subsequent makeup generation to ensure that the generated makeup meets the user's specific color type. Subsequently, based on the generated color information, the user's makeup image is generated through a two-stage diffusion model. In the forward diffusion process, the non-makeup image is gradually converted into a makeup image through noise, and the noise is gradually removed in the reverse diffusion process to generate the final makeup image, and the makeup generation that meets the user's personalized needs is achieved through the conditional diffusion model.

[0073] Furthermore, the present invention uses a variational lower bound (VLB) optimization model, and the formula is as follows:

[0074]

[0075] The variational lower bound is used to optimize the diffusion model so that it generates high-quality personalized images during the denoising process.

[0076] To train the above model, the present invention collected 32,026 makeup images of 200 users and preprocessed the facial image datasets with and without makeup. To ensure the diversity of the collected dataset, samples of different ages and genders were included, with an age distribution of 17-28 years old and a male-female ratio of 2:8.

[0077] The training process is as follows:

[0078] The user's selfie image is input into the diffusion model, and the personal color analysis results are input as conditional information. Face detection and landmark point extraction are performed on the user's selfie face image without makeup to ensure that the facial area in the image is aligned. The image is standardized to meet the model input requirements. The personal color analysis results (iris, hair and skin color information) obtained by the preprocessing module are used. This color information is embedded into the feature representation of the input image as conditional information to guide the makeup generation process.

[0079] Under the guidance of conditional information, the two-stage diffusion model generates images with personalized makeup. The personal color analysis results are input into the diffusion model as conditional information, and the conditional generative adversarial network (Conditional GANs) or conditional variational autoencoder (Conditional VAEs) are used to implement the above guided makeup generation process. Starting from a pure noise image, the target makeup image x is generated by step-by-step denoising. T Each step of the denoising process is guided by conditional information to ensure that the generated results meet the user's personalized needs. During the training process, user feedback is used to adjust the model parameters and continuously optimize the generation effect. Mixed precision training and model compression technology can also be used to improve the model's inference speed and efficiency.

[0080] Step 3: Perform refinement and optimization post-processing on the generated facial makeup image.

[0081] In this step, a makeup face image post-processing model is designed. By applying image processing technology, the generated makeup image is refined and optimized to ensure the naturalness and high quality of the generated results. The implementation methods of refinement post-processing and optimization post-processing are as follows:

[0082] Step 31, performing Gaussian Blur processing on the generated facial makeup image.

[0083] This step uses Gaussian blur to smooth the image and remove small noises to make the makeup effect more natural. Gaussian blur is achieved by applying a Gaussian kernel to the image for convolution.

[0084] The formula for the Gaussian kernel is:

[0085]

[0086] Here, σ is the standard deviation of the Gaussian kernel.

[0087] Step 32, edge detection is performed on the Gaussian blurred image to enhance the edge features of the image, strengthen the details of the makeup area, ensure that the makeup lines are clearer and more natural, and make the makeup area clearer, so as to obtain a preliminary makeup image I makeup .

[0088] This step enhances the edge features of the image through edge detection to make the makeup area clearer. Specifically, the Canny edge detection algorithm can be used, and the Sobel operator can be used to calculate the gradient strength and direction of the image. Non-maximum suppression is performed on the gradient image, and the local maximum value is retained as the edge. High and low double thresholds are applied to connect and filter the edges. The mathematical formulas of Canny edge detection include:

[0089] Sobel operator:

[0090]

[0091] Gradient strength and direction:

[0092]

[0093] In general, the naturalness and details of the makeup area are enhanced through the above-mentioned Gaussian blur and smoothing, edge detection and other technologies.

[0094] Step 33: The preliminary makeup image I makeup It is then fused with the user's selfie face image without makeup (ImageBlending), and the natural skin color and facial details are retained through a weighted algorithm to obtain the final makeup image I blend , the fusion formula is expressed as follows:

[0095] I blend =α·I original +(1-α)·I makeup

[0096] Among them, α is the fusion weight coefficient, I original It is a selfie of the user’s face without makeup.

[0097] In addition, the present invention also designs a user feedback mechanism, where users can provide feedback based on the generated makeup effects. The system will dynamically adjust the generation model and optimize the generation parameters or regenerate the makeup based on these feedback data, further improving the personalization and naturalness of the makeup effects. Specifically, the user interface design is used to provide users with an intuitive feedback interface, allowing users to evaluate and adjust the generated makeup effects. The user interface can be designed using a Web framework (Vue.js). Functions include a rating system, a feedback input box, and a slider for adjusting makeup parameters. Collect user satisfaction and improvement suggestions for the generated makeup. Send user feedback data to the server for storage and analysis. Data types include user ratings, opinions and suggestions, and data for adjusting parameters.

[0098] The user's feedback on the generated makeup is collected through the user interface, and the generation parameters are adjusted or the makeup is regenerated based on the feedback data. The optimization goal of the generation parameters is:

[0099]

[0100] Where F is the user feedback data, η is the learning rate, and L is the loss function, which measures the gap between the generated result and the user's expectation. When the diffusion model generates the user's makeup image, it will generate iteratively. When generating the makeup image twice, the target is the newly generated user makeup image parameter θ new The quality of is higher than that of the user makeup image parameter θ generated in the previous round old , where θ old Represents θ new The makeup image parameters generated previously are generally referred to as θ. All makeup image parameters are iteratively optimized and generated during the diffusion model training process. The present invention can adjust the generation parameters according to user feedback to optimize the makeup effect. The user feedback data is analyzed to automatically adjust the generation parameters of the model or regenerate the makeup image. The generation parameters are adjusted using a machine learning algorithm (such as stochastic gradient descent SGD).

[0101] The staged generation method of the present invention can better control the details and overall effects of the generation process, and improve the naturalness and realism of the generated image. Although the diffusion model has advantages in image generation quality, its computational complexity is high, resulting in a slow generation speed, which is difficult to meet the needs of real-time applications. In order to achieve real-time generation of facial makeup, comprehensive improvements must be made in model optimization, reasoning acceleration, and system architecture design. Therefore, in order to improve the real-time performance of the system, the present invention also integrates the trained two-stage diffusion model and adversarial network into the TensorRT engine, optimizes the model to improve reasoning speed and efficiency, and allows makeup generation to be performed in real time.

[0102] Specifically, the present invention uses a deep learning framework (PyTorch) to train a model for generating makeup images. The trained model is usually saved as a model file (.pt). The trained model is converted to ONNX format, which is an open model format that can interoperate between different deep learning frameworks. This step ensures that the model can be read and processed by TensorRT. The ONNX model is converted to a TensorRT engine using TensorRT's API. During the conversion process, TensorRT applies a variety of optimization techniques, including network layer fusion, kernel automatic tuning, and precision mixing (FP32, FP16, INT8) to maximize the inference speed and efficiency of the model. The model optimized by TensorRT is used for real-time inference on the GPU to generate makeup images. A TensorRT engine and inference context are created to perform inference tasks. The engine contains the optimized network structure and weights, and the context is used to manage the resources required in the inference process. GPU memory is allocated for model input and output, and input data (user selfie images and personal color analysis results) are uploaded to the GPU. The input data needs to be preprocessed, including image standardization and color information embedding. The inference process is performed using the TensorRT engine to generate images with personalized makeup from the input data. During the inference process, the model generates makeup effects that meet the user's needs based on the input conditional information (such as the user's personalized color, age, gender, etc.). The generated images are post-processed, including the application of Gaussian blur, edge detection and other technologies to ensure the naturalness and high quality of the generated results. The generated results are then displayed through the user interface, and a feedback mechanism is provided. The best computing kernel is automatically selected based on the specific hardware configuration and input data to achieve efficient computing. Using INT8 (8-bit integer) precision inference, the computing requirements of the model are reduced through quantization, and the inference speed is increased.

[0103] In summary, the present invention uses facial key points to extract the color information of the iris, hair and skin, and determines the user's personal color type through color conversion and clustering methods; then uses a two-stage diffusion model to generate images with personalized makeup in real time under the guidance of conditional information, and the user's personal color, age, gender and makeup purpose can be considered in the generation process; after the makeup face image is generated, image processing is performed to refine and optimize the generated makeup image, and the user's feedback on the generated makeup is collected through the user interface, and the generation parameters are adjusted or the makeup is regenerated according to the feedback. Based on the above content, customized makeup can be generated for users in real time.

[0104] The following two specific examples are used to illustrate the dimensionality reduction method and the overall execution process of the above method.

[0105] This embodiment is built on a cloud computing platform, which consists of 15 servers, including VMware Esxi5, 20T disk array and 1000M network switch. All embodiments are deployed on virtual machines.

[0106] Figure 2 This is a facial makeup image generated by the network trained using the dataset collected by the present invention, including lip gloss, beauty contact lenses, foundation, eye shadow, double eyelids, eyebrows, etc. The left side of the face shows the effect after makeup, and the right side shows the user's original bare face state. This shows the natural fusion of the makeup effect generated by the system and the user's original facial features. In the makeup area on the left, you can see that the system has generated makeup details such as eye shadow, eyebrows, and lip makeup, and the colors match the user's personalized color information, reflecting the system's ability to generate personalized makeup. This image clearly shows how the makeup generation model of the present invention adds natural makeup to the user's face while maintaining the natural skin color and details of the face.

[0107] Figure 3 This is the image effect of makeup generated on an existing foreign open source dataset using the method of the present invention. Among them, the upper row shows the guiding conditions, that is, different types of makeup styles, and the lower row shows the results generated according to these conditions. Starting from the far left, the original image is processed to generate makeup effects of different styles. These makeup styles cover a variety of types from fresh and light makeup to heavy and bright makeup. Each generated makeup well reflects the guiding style of the upper row, while also maintaining the features and details of the user's original face. This demonstrates the high adaptability and diversity of the present invention in achieving makeup migration.

[0108] The above image shows the makeup effects generated by the real-time facial makeup generation method based on the two-stage diffusion model. The experimental results show that the method can efficiently generate a variety of different styles of makeup on the original face image, and the effect is natural and diverse. As can be seen from the figure, the generated makeup effects cover a variety of styles, including natural light makeup, heavy makeup, and various fashion makeup. This shows that the model has strong diversity and adaptability in generating makeup, and can meet the needs of different users. The generated makeup image blends very naturally with the original image, without obvious flaws or inconsistencies. The application of image optimization techniques such as Gaussian blur and edge detection further enhances the naturalness of the makeup effect, making the generated image look more realistic. The model performs well in detail processing and can accurately add color and details to each makeup part such as eye shadow, lipstick, and blush. Whether it is the delicate depiction of the eyes or the color rendering of the lips, it shows the model's superb processing ability in details. By adding the user's personalized color analysis results, the makeup effects generated by the model can well adapt to the skin color, iris color, and hair color of different users. This not only enhances the user experience, but also improves the personalization and uniqueness of the generated images. Using TensorRT's optimization technology, this method has been significantly improved in inference speed, achieving real-time generation of makeup images. This is of great significance for practical application scenarios such as virtual makeup trials, live broadcasts, and social media.

[0109] Although the present invention has been described through preferred embodiments, the present invention is not limited to the embodiments described herein but includes various changes and modifications that may be made without departing from the scope of the present invention.

Claims

1. A real-time facial makeup generation method based on a two-stage diffusion model, characterized in that: The steps include: Step 1, preprocessing the user's face image without makeup to analyze the color and contour features of the user's face, obtain personal color information of different users, and classify them into predefined color types, that is, personal color analysis results of different users; Step 2: Based on the two-stage diffusion model, a facial makeup image is generated according to the bare face image uploaded by the user and the personal color analysis result; Step 3, performing refinement post-processing and optimization post-processing on the generated facial makeup image.

2. According to claim 1, the real-time generation method of facial makeup based on the two-stage diffusion model is characterized in that: The implementation method of step 1, face preprocessing is as follows: Step 11, detecting the user's plain face image to obtain a face area; Step 12, extracting average color values ​​from the iris area, hair area, and skin area in the face area, respectively, and converting the RGB color space into the Lab color space to extract personalized color data, that is, personal color information of different users; Step 13: cluster the Lab color space information of the iris area, the hair area, and the skin area to obtain different predefined color types.

3. According to claim 1, the real-time generation method of facial makeup based on the two-stage diffusion model is characterized in that: In step 11, n user's bare face images are detected to obtain n face regions, and the average color values ​​are extracted from the iris region, hair region and skin region of the i-th user, and converted into Lab color space information respectively. and Represented as x i ; In step 13, K-means clustering is used, with μ j is the cluster center, and the objective function J is: Where k is the number of personal color types, r ij is an indicator variable, when the sample x i It is 1 if it belongs to the jth cluster, otherwise it is 0.

4. According to claim 3, the real-time generation method of facial makeup based on the two-stage diffusion model is characterized in that: The cluster centers are pre-set according to the warm and cool tones and the season types.

5. According to claim 1, the real-time generation method of facial makeup based on the two-stage diffusion model is characterized in that: In step 2, the two-stage diffusion model generates a facial makeup image by a stepwise denoising method, that is, in the forward diffusion process, the plain face image x0 is gradually converted into a pure noise image by adding noise, and in the reverse diffusion process, the pure noise image is gradually converted into the target makeup image x by removing noise and combining the personal color analysis results. T , that is, facial makeup image.

6. The real-time generation method of facial makeup based on the two-stage diffusion model according to claim 5 is characterized in that: The forward diffusion process converts the plain face image x0 into a noisy image x t , the implementation method is as follows: in, is a parameter that controls the amount of noise, t is the time step, I is the identity matrix, q(x t |x0) represents the probability distribution of the data point at time t given the initial data point x0; is a Gaussian distribution, where x t At the data point at time t, the distribution is is the mean value, is the covariance matrix; The reverse diffusion process converts the pure noise image into the target makeup image x T , the implementation method is as follows: p θ (x t-1 ∣x t ,c)=N(x t-1 ;μ θ (x t ,t,c),Σ θ (x t ,t,c)) Among them, μ θ and Σ θ is a learnable parameter, c represents the user’s personal color analysis result, and guides the model to generate makeup that meets personalized needs.

7. The real-time facial makeup generation method based on the two-stage diffusion model according to claim 6 is characterized in that: When one of the following conditions is met, the forward diffusion process is terminated: Condition 11: When t is equal to the preset fixed time step number T, the data has been covered by enough noise, and all data can evenly reach the state of high noise; Condition 12, when the noise level reaches a set threshold, the threshold is sufficient to make any significant difference between the original data and the noise unrecognizable; When one of the following conditions is met, the reverse diffusion process is terminated: Condition 21, starting from the time step when the forward diffusion process terminates, the data is gradually restored to time step 0. When the time step reaches 0, the output is regarded as the restored clean data; Condition 22: When the makeup data quality reaches a preset threshold, for example, the clarity is not less than 3% of the original image, and the error is less than 0.

01.

8. The real-time facial makeup generation method based on the two-stage diffusion model according to claim 1 is characterized in that: The implementation method of step 3, refining post-processing and optimizing post-processing is as follows: Step 31, performing Gaussian blur on the generated facial makeup image; Step 32, edge detection is performed on the Gaussian blurred image to enhance the edge features of the image, making the makeup area clearer, and obtaining a preliminary makeup image I makeup ; Step 33: The preliminary makeup image I makeup It is merged with the user's selfie without makeup to retain the natural skin color and details, and the final makeup image I is obtained blend .

9. The real-time facial makeup generation method based on the two-stage diffusion model according to claim 1 is characterized in that: In step 3, user feedback on the generated makeup is collected through the user interface, and the generation parameters are adjusted or the makeup is regenerated according to the feedback data, wherein the optimization goal of the generation parameters is: Where F is the user feedback data, η is the learning rate, and L is the loss function. When the diffusion model generates the user makeup image, it will iteratively generate it. When generating the makeup image twice, the target is the newly generated user makeup image parameter θ new The quality of is higher than that of the user makeup image parameter θ generated in the previous round old .

10. The real-time facial makeup generation method based on the two-stage diffusion model according to claim 1 is characterized in that: Integrate the trained two-stage diffusion model and adversarial network into the TensorRT engine and optimize the model to improve inference speed and efficiency as follows: Use a deep learning framework to train a model for generating makeup images, save the trained model as a model file, and then convert it to ONNX format. Use the TensorRT API to convert the ONNX format model to the TensorRT engine and create an inference context. The TensorRT-optimized model is used to perform real-time inference on the GPU to generate makeup images. The TensorRT engine includes the optimized network structure and weights, and the inference context is used to manage the resources required during the inference process.

Images