Face makeup transfer method based on improved diffusion model

Abstract

The application discloses a face makeup transfer method based on an improved Diffusion model, comprising: preliminary data processing, obtaining a pseudo makeup dataset, the dataset is made by using the public MT dataset, the LADN dataset and the Wild-MT dataset by using histogram matching combined with a thin plate spline interpolation method; training the improved Diffusion model by using the obtained dataset to obtain an optimal makeup transfer model; the loss function of model training comprises a latent diffusion loss, a makeup loss and a diffusion reconstruction loss, the latent diffusion loss is used for guiding the model to learn to gradually restore a clear face makeup transfer image from a noise image, the makeup loss is used for optimizing the makeup effect of the generated image, and the diffusion reconstruction loss is used for keeping the consistency of the image; the optimal makeup transfer model is applied to process input source images and reference images to generate makeup transfer results with natural transition. The application can realize a high-quality and controllable makeup transfer effect while keeping the identity features of the source images.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision and image processing, and particularly discloses a face makeup transfer method based on an improved Diffusion model. BACKGROUND

[0002] Makeup transfer aims to transfer the makeup style in the reference image to the source image while preserving the original identity features of the source image. Due to the diversity of makeup styles in the reference image, how to accurately map these styles to the source image is a challenging task. Existing makeup transfer methods mainly rely on generative adversarial networks to synthesize makeup images. These methods have the following problems: first, most methods still regard makeup style as a change in color distribution, ignoring makeup details; second, these methods tend to pixel-level conversion, and when the face shape difference between the source image and the target image is large, the effect is limited; third, generative adversarial network models are prone to mode collapse when training high-resolution images, resulting in a lack of diversity in the generated results.

[0003] Diffusion model has attracted much attention due to its excellent generation ability. Compared with generative adversarial networks, Diffusion model shows better stability in the training process and is good at generating images with fine realistic details. The iterative refinement feature of Diffusion model makes it particularly suitable for makeup transfer tasks. However, how to control the quality of the generated results is still a challenge. SUMMARY

[0004] The present application aims to overcome the shortcomings and deficiencies of the prior art and provides a face makeup transfer method based on an improved Diffusion model, which can solve the problems of existing methods that only focus on pixel-level conversion, ignore makeup details, are unstable in training, and lack control ability in generating results, and achieve high-quality and controllable makeup transfer effect while preserving the identity features of the source image.

[0005] To achieve the above objectives, the technical solution provided by this invention is as follows: a face makeup transfer method based on an improved Diffusion model. This improved Diffusion model modifies the backbone network module of the original Diffusion model and adds a facial component perception and matching module. The improvement to the backbone network module involves innovatively defining the makeup transfer problem as a repair task and constructing a makeup repair module to reduce initial Gaussian noise introduced during sampling and mitigate randomness introduced during repair. This makeup repair module occludes the area to be covered in the source image and repairs the occluded portion of the source image based on a reference image to achieve makeup transfer. Simultaneously, it introduces… The DINO-V2 pre-trained model is used as a feature extractor to encode reference images, providing reliable feature correspondences for makeup transfer and enhancing the pixel-level control of the model. The facial part perception matching module fuses feature maps of the source and reference images through spatial alignment. It includes a lightweight makeup encoder and a spatially perceptive makeup fusion unit. The makeup encoder generates feature maps of different resolutions through feature extractors and downsampling. The spatially perceptive makeup fusion unit captures makeup details and global information based on feature maps of different resolutions, realizing spatial alignment and fusion of feature maps of the source and reference images, and injects the fused feature map into the intermediate layer of the U-Net network of the improved Diffusion model.

[0006] The specific implementation of this facial makeup transfer method includes the following steps:

[0007] 1) Preliminary data processing to obtain a pseudo-makeup dataset, which is created by using the publicly available MT dataset, LADN dataset and Wild-MT dataset using histogram matching combined with thin plate spline interpolation method;

[0008] 2) Use the dataset obtained in step 1) to train the improved Diffusion model, and obtain the trained improved Diffusion model as the optimal makeup transfer model; wherein, the loss function for model training includes latent diffusion loss, makeup loss and diffusion reconstruction loss, the latent diffusion loss is used to guide the model to learn to gradually recover clear facial makeup transfer images from noisy images, the makeup loss is used to optimize the makeup effect of the generated images, and the diffusion reconstruction loss is used to maintain the consistency of the images;

[0009] 3) Apply the optimal makeup transfer model to process the input source and reference images to generate makeup transfer results with natural transitions.

[0010] Furthermore, the makeup transfer problem is defined as a restoration task, and a makeup restoration module is constructed, which performs the following process: First, for the source image x and the reference image y, the occlusion part m of the restoration region is constructed, and the occluded face image x is obtained. m Then the face image xm The improved Diffusion model is input into the U-Net network for iterative denoising. The training of the U-Net network includes a forward process and a backward process: in the forward process, Gaussian noise ∈ is gradually added to the initial feature map z0 to generate the feature map z at time t. t , means as follows:

[0011]

[0012] In the formula, z0 represents the face images I in the fake makeup dataset generated using a pre-trained encoder. w The initial feature map obtained from encoding, where s is the time index, and α... s =1-β s and β s This indicates the degree to which the original face image is preserved and the noise scheduling parameters. This represents the overall proportion of the initial feature map z0 retained at time t; during the reverse process, the added noise ∈ is predicted using the U-Net network.

[0013] Furthermore, for the lightweight makeup encoder, the source and reference images, after being rearranged into feature maps F0, are first processed by the first feature extractor V1. V1 contains a convolutional layer and two residual blocks to generate feature map F1. Subsequently, each feature map enters the next feature extractor V1. i The previous downsampling operation generates the subsequent feature map F. i Where i∈2,3,4; four feature extractors progressively process and downsample the feature map F of the previous layer. i-1 The generation process is represented as follows:

[0014] F1 = V1(F0)

[0015] F i =V i (DS(F i-1 ), i∈2,3,4

[0016] In the formula, DS represents the downsampling block, and F i It is a set of four feature maps, each with dimensions aligned with the intermediate layers within the U-Net network;

[0017] Considering that feature maps of different resolutions have varying capabilities in capturing makeup details and global information, a spatially aware makeup fusion unit is introduced to fuse feature maps from source and reference images of different resolutions; the feature map F extracted by the aforementioned lightweight makeup encoder is then used... i Decomposed into feature maps F of the source image along the batch dimension i x Feature map F of the reference image iy And input into the spatially aware makeup fusion unit; for high-resolution feature maps, the feature map F of the reference image is first matched by feature point matching. i y Feature map F aligned to the source image i x , denoted as F i y,x Then F i y,x and F i x It is connected to the corresponding feature point embedding; for low-resolution features, feature point matching is sufficient to establish F. i y and F i x The pixel correspondence between features ensures global consistency; the above process is expressed as:

[0018]

[0019] In the formula, It is a feature map connected with location embedding to incorporate spatial information; location embedding integrates facial feature point information into the feature map; This represents a Transformer based on cross-attention; the feature map size is 64 when i=1, 32 when i=2, 16 when i=3, and 8 when i=4; at each scale, the feature map fused by the Transformer mechanism based on cross-attention is... Added to the intermediate layer of the U-Net network for precise control over generation.

[0020] Furthermore, in step 2), a supervised learning approach is used to train an improved Diffusion model for facial makeup transfer. The input is the source image and the reference image, and the output is the processed source image with the reference facial makeup.

[0021] The loss function for training the improved Diffusion model consists of three parts: latent diffusion loss, makeup loss, and diffusion reconstruction loss. The latent diffusion loss is used to optimize the denoising process of the improved Diffusion model. It optimizes the model by minimizing the mean square error between the predicted noise and the actual noise. The calculation formula is as follows:

[0022]

[0023] In the formula, e x,y This represents the concatenation of the source and reference images along the channel dimension; This represents the predicted noise of the makeup repair module. This refers to the DINO-V2 pre-trained model. This indicates the parameters of the facial component perception and matching module;

[0024] The makeup loss is used to optimize the makeup effect of the generated image, and the calculation formula is as follows:

[0025]

[0026] In the formula, I0 is the image generated using a pre-trained decoder, and Ψ(·,·) represents the pseudo-truth function, which is the image of the pseudo-makeup dataset made by using histogram matching combined with thin plate spline interpolation.

[0027] The diffusion reconstruction loss is used to maintain image consistency, aiming to ensure that the generated output remains similar to the source image when the source image is used as a reference image. This diffusion reconstruction loss encourages the improved diffusion model to retain the inherent features and details of the source image, which is beneficial for more stable output and ensures visually consistent results. The calculation formula is as follows:

[0028]

[0029] In the formula, This indicates a connection along the spatial dimension between the encoded source image, the encoded source masking image, and the downsampled source masking image; e x,x They are two identical source images connected along the channel dimension; This indicates that the coded reference image, the coded reference occlusion image, and the downsampled reference occlusion image are connected along the spatial dimension; e y,y These are two identical reference images connected along the channel dimension; therefore, the optimization problem of makeup transfer is formulated as follows:

[0030]

[0031] In the formula, θ1, θ2, and θ3 represent the training parameters of the makeup repair module, the DINO-V2 pre-trained model, and the facial component perception and matching module, respectively; λ ldm and λ drec These are the weighting coefficients for potential diffusion loss and diffusion reconstruction loss, respectively.

[0032] Furthermore, in step 3), the specific process of applying the trained optimal makeup transfer model is as follows: First, given the face data to be transferred, including the corresponding source image and reference image, the encoded source image, encoded source occlusion image and downsampled source occlusion image connected along the spatial dimension are obtained through data preprocessing, and are used together as the model input. The output is the makeup transfer result with natural transition.

[0033] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0034] 1. This invention achieves makeup transfer by improving the Diffusion model, generating more natural and realistic results. It is more reliable than generative adversarial network methods that rely solely on pixel-level transformations and suffer from unstable training.

[0035] 2. This invention redefines the makeup migration problem as a repair task, which reduces the complexity of the problem, allows for a greater focus on generating makeup details, reduces artifacts that may result from global generation, and improves generation quality.

[0036] 3. This invention has strong generalization ability and can be applied to the transfer of different makeup styles.

[0037] 4. The improved diffusion reconstruction loss used in the training of the Diffusion model in this invention can further enhance the inherent features and details of face images, producing more stable output and visually coherent results.

[0038] 5. The facial component perception and matching module introduced in this invention can achieve precise feature matching. The spatial perception makeup fusion unit ensures accurate transfer of makeup details. Attached Figure Description

[0039] Fig. 1 This is a schematic diagram of the logic flow of the present invention.

[0040] Fig. 2 This is a schematic diagram of the facial component perception and matching module.

[0041] Fig. 3 A schematic diagram of the structure for improving the Diffusion model. Detailed Implementation

[0042] The present invention will be further described in detail below with reference to the embodiments and accompanying drawings, but the embodiments of the present invention are not limited thereto.

[0043] like Figs. 1 to 3As shown, this embodiment discloses a face makeup transfer method based on an improved Diffusion model. This improved Diffusion model modifies the backbone network module of the original Diffusion model and adds a facial part perception and matching module. The improvement to the backbone network module involves innovatively defining the makeup transfer problem as a repair task and constructing a makeup repair module to reduce initial Gaussian noise introduced during sampling and mitigate randomness introduced during repair. This makeup repair module occludes the area to be covered in the source image and repairs the occluded portion of the source image based on a reference image to achieve makeup transfer. Simultaneously, DINO is introduced... - The V2 pre-trained model is used as a feature extractor to encode the reference image, providing a reliable feature correspondence for makeup transfer and enhancing the pixel-level control of the model. The facial part perception matching module fuses the feature maps of the source image and the reference image through spatial alignment. It includes a lightweight makeup encoder and a spatially perceptive makeup fusion unit. The makeup encoder generates feature maps of different resolutions through feature extractor and downsampling. The spatially perceptive makeup fusion unit captures makeup details and global information according to the feature maps of different resolutions, realizes the spatial alignment and fusion of the feature maps of the source image and the reference image, and injects the fused feature map into the intermediate layer of the U-Net network of the improved Diffusion model.

[0044] The specific implementation of this makeup transfer method includes the following steps:

[0045] 1) Preliminary data processing to obtain a pseudo-makeup dataset, which is created by using the publicly available MT dataset, LADN dataset and Wild-MT dataset using histogram matching combined with thin plate spline interpolation.

[0046] 2) Use the dataset obtained in step 1) to train the improved Diffusion model, and obtain the trained improved Diffusion model as the optimal makeup transfer model; wherein, the loss function for model training includes latent diffusion loss, makeup loss and diffusion reconstruction loss, the latent diffusion loss is used to guide the model to learn to gradually recover clear facial makeup transfer images from noisy images, the makeup loss is used to optimize the makeup effect of the generated images, and the diffusion reconstruction loss is used to maintain the consistency of the images;

[0047] The makeup transfer problem is defined as a restoration task. A makeup restoration module is constructed, which performs the following process: First, for the source image x and the reference image y, the occlusion part m of the restoration region is constructed, and the occluded face image x is obtained. m Then the face image x mThe improved Diffusion model is input into the U-Net network for iterative denoising. The training of the U-Net network includes a forward process and a backward process: in the forward process, Gaussian noise ∈ is gradually added to the initial feature map z0 to generate the feature map z at time t. t , means as follows:

[0048]

[0049] In the formula, z0 represents the face images I in the fake makeup dataset generated using a pre-trained encoder. w The initial feature map obtained from encoding, where s is the time index, and α... s =1-β s and β s This indicates the degree to which the original face image is preserved and the noise scheduling parameters. This represents the overall proportion of the initial feature map z0 retained at time t; during the reverse process, the added noise ∈ is predicted using the U-Net network.

[0050] For the lightweight makeup encoder, the source and reference images, after being rearranged into feature maps F0, are first processed by the first feature extractor V1. V1 contains a convolutional layer and two residual blocks to generate feature map F1. Subsequently, each feature map enters the next feature extractor V1. i The previous downsampling operation generates the subsequent feature map F. i Where i∈2,3,4; four feature extractors progressively process and downsample the feature map F of the previous layer. i-1 The generation process is represented as follows:

[0051] F1 = V1(F0)

[0052] F i =V i (DS(F i-1 ), i∈2,3,4

[0053] In the formula, DS represents the downsampling block, and F i It is a set of four feature maps, each with dimensions aligned with the intermediate layers within the U-Net network;

[0054] Considering that feature maps of different resolutions have varying capabilities in capturing makeup details and global information, a spatially aware makeup fusion unit is introduced to fuse feature maps from source and reference images of different resolutions; the feature map F extracted by the aforementioned lightweight makeup encoder is then used... i Decomposed into feature maps F of the source image along the batch dimension i x Feature map F of the reference image i yAnd input into the spatially aware makeup fusion unit; for high-resolution feature maps, the feature map F of the reference image is first matched by feature point matching. i y Feature map F aligned to the source image i x , denoted as F i y,x Then F i y,x and F i x It is connected to the corresponding feature point embedding; for low-resolution features, feature point matching is sufficient to establish F. i y and F i x The pixel correspondence between features ensures global consistency; the above process is expressed as:

[0055]

[0056] In the formula, It is a feature map connected with location embedding to incorporate spatial information; location embedding integrates facial feature point information into the feature map; This represents a Transformer based on cross-attention; the feature map size is 64 when i=1, 32 when i=2, 16 when i=3, and 8 when i=4; at each scale, the feature map fused by the Transformer mechanism based on cross-attention is... Added to the intermediate layer of the U-Net network for precise control over generation.

[0057] An improved Diffusion model for facial makeup transfer is trained using supervised learning. The input is a source image and a reference image, and the output is a processed source image with the reference facial makeup.

[0058] The loss function for training the improved Diffusion model consists of three parts: latent diffusion loss, makeup loss, and diffusion reconstruction loss. The latent diffusion loss is used to optimize the denoising process of the improved Diffusion model. It optimizes the model by minimizing the mean square error between the predicted noise and the actual noise. The calculation formula is as follows:

[0059]

[0060] In the formula, e x,y This represents the concatenation of the source and reference images along the channel dimension; This represents the predicted noise of the makeup repair module. This refers to the DINO-V2 pre-trained model. This indicates the parameters of the facial component perception and matching module;

[0061] The makeup loss is used to optimize the makeup effect of the generated image, and the calculation formula is as follows:

[0062]

[0063] In the formula, I0 is the image generated using a pre-trained decoder, and Ψ(·,·) represents the pseudo-truth function, which is the image of the pseudo-makeup dataset made by using histogram matching combined with thin plate spline interpolation.

[0064] The diffusion reconstruction loss is used to maintain image consistency, aiming to ensure that the generated output remains similar to the source image when the source image is used as a reference image. This diffusion reconstruction loss encourages the improved diffusion model to retain the inherent features and details of the source image, which is beneficial for more stable output and ensures visually consistent results. The calculation formula is as follows:

[0065]

[0066] In the formula, This indicates a connection along the spatial dimension between the encoded source image, the encoded source masking image, and the downsampled source masking image; e x,x They are two identical source images connected along the channel dimension; This indicates that the coded reference image, the coded reference occlusion image, and the downsampled reference occlusion image are connected along the spatial dimension; e y,y These are two identical reference images connected along the channel dimension; therefore, the optimization problem of makeup transfer is formulated as follows:

[0067]

[0068] In the formula, θ1, θ2, and θ3 represent the training parameters of the makeup repair module, the DINO-V2 pre-trained model, and the facial component perception and matching module, respectively; λ ldm and λ drec These are the weighting coefficients for potential diffusion loss and diffusion reconstruction loss, respectively.

[0069] 3) The optimal makeup transfer model is applied to process the input source and reference images to generate makeup transfer results with natural transitions. The specific process is as follows:

[0070] First, given the face data to be transferred (including the corresponding source image and reference image), the data preprocessing obtains the encoded source image, encoded source occlusion image and downsampled source occlusion image connected along the spatial dimension, and uses them together as the model input. The output is the makeup transfer result with a natural transition.

[0071] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A face makeup transfer method based on an improved Diffusion model, characterized in that, This improved Diffusion model enhances the backbone network module of the original Diffusion model and adds a facial part perception and matching module. The improvement to the backbone network module involves innovatively defining the makeup transfer problem as a restoration task and constructing a makeup restoration module. This module reduces initial Gaussian noise introduced during sampling and mitigates randomness introduced during restoration. It masks the areas to be covered in the source image and restores the masked portions of the source image based on a reference image to achieve makeup transfer. Simultaneously, a DINO-V2 pre-trained model is introduced as a feature extractor to encode parameters. The reference image provides a reliable feature correspondence for makeup transfer to enhance the pixel-level control of the model; the facial part perception matching module fuses the feature maps of the source image and the reference image through spatial alignment, which includes a lightweight makeup encoder and a spatially perceptive makeup fusion unit. The makeup encoder generates feature maps of different resolutions through feature extractor and downsampling. The spatially perceptive makeup fusion unit captures makeup details and global information according to the feature maps of different resolutions, realizes the spatial alignment and fusion of the feature maps of the source image and the reference image, and injects the fused feature map into the intermediate layer of the U-Net network of the improved Diffusion model; The specific implementation of this facial makeup transfer method includes the following steps: 1) Preliminary data processing to obtain a pseudo-makeup dataset. This dataset was created using the publicly available MT dataset, LADN dataset, and Wild-MT dataset, employing histogram matching combined with thin-plate spline interpolation. 2) Use the dataset obtained in step 1) to train the improved Diffusion model, and obtain the trained improved Diffusion model as the optimal makeup transfer model; wherein, the loss function for model training includes latent diffusion loss, makeup loss and diffusion reconstruction loss, the latent diffusion loss is used to guide the model to learn to gradually recover clear facial makeup transfer images from noisy images, the makeup loss is used to optimize the makeup effect of the generated images, and the diffusion reconstruction loss is used to maintain the consistency of the images; 3) Apply the optimal makeup transfer model to process the input source and reference images to generate makeup transfer results with natural transitions.

2. The face makeup transfer method based on the improved Diffusion model according to claim 1, characterized in that, The makeup transfer problem is defined as a restoration task. A makeup restoration module is constructed, which performs the following process: First, for the source image x and the reference image y, the occlusion part m of the restoration region is constructed, and the occluded face image x is obtained. m Then the face image x m The improved Diffusion model is input into the U-Net network for iterative denoising. The training of the U-Net network includes a forward process and a backward process: in the forward process, Gaussian noise ∈ is gradually added to the initial feature map z0 to generate the feature map z at time t. t , means as follows: In the formula, z0 represents the face images I in the fake makeup dataset generated using a pre-trained encoder. w The initial feature map obtained from encoding, where s is the time index, and α... s =1-β s and β s This indicates the degree to which the original face image is preserved and the noise scheduling parameters. This represents the overall proportion of the initial feature map z0 retained at time t; during the reverse process, the added noise ∈ is predicted using the U-Net network.

3. The face makeup transfer method based on the improved Diffusion model according to claim 2, characterized in that, For the lightweight makeup encoder, the source and reference images, after being rearranged into feature maps F0, are first processed by the first feature extractor V1. V1 contains a convolutional layer and two residual blocks to generate feature map F1. Subsequently, each feature map enters the next feature extractor V1. i The previous downsampling operation generates the subsequent feature map F. i Where i∈2,3,4; four feature extractors progressively process and downsample the feature map F of the previous layer. i-1 The generation process is represented as follows: F1 = V1(F0) F i =V i (DS(F i-1 )),iv2,3,4 In the formula, DS represents the downsampling block, and F i It is a set of four feature maps, each with dimensions aligned with the intermediate layers within the U-Net network; Considering that feature maps of different resolutions have varying capabilities in capturing makeup details and global information, a spatially aware makeup fusion unit is introduced to fuse feature maps from source and reference images of different resolutions; the feature map F extracted by the aforementioned lightweight makeup encoder is then used... i Decomposed into feature maps F of the source image along the batch dimension i x Feature map F of the reference image i y And input into the spatially aware makeup fusion unit; for high-resolution feature maps, the feature map F of the reference image is first matched by feature point matching. i y Feature map F aligned to the source image i x , denoted as F i y,x Then F i y,x and F i x It is connected to the corresponding feature point embedding; for low-resolution features, feature point matching is sufficient to establish F. i y and F i x The pixel correspondence between features ensures global consistency; the above process is expressed as: In the formula, It is a feature map connected with location embedding to incorporate spatial information; location embedding integrates facial feature point information into the feature map; This represents a Transformer based on cross-attention; the feature map size is 64 when i=1, 32 when i=2, 16 when i=3, and 8 when i=4; at each scale, the feature map fused by the Transformer mechanism based on cross-attention is... Added to the intermediate layer of the U-Net network for precise control over generation.

4. The face makeup transfer method based on the improved Diffusion model according to claim 3, characterized in that, In step 2), a supervised learning approach is used to train an improved Diffusion model for facial makeup transfer. The input is the source image and the reference image, and the output is the processed source image with the reference facial makeup. The loss function for training the improved Diffusion model consists of three parts: latent diffusion loss, makeup loss, and diffusion reconstruction loss. The latent diffusion loss is used to optimize the denoising process of the improved Diffusion model. It optimizes the model by minimizing the mean square error between the predicted noise and the actual noise. The calculation formula is as follows: In the formula, e x,y This represents the concatenation of the source and reference images along the channel dimension; This represents the predicted noise of the makeup repair module. This refers to the DINO-V2 pre-trained model. This indicates the parameters of the facial component perception and matching module; The makeup loss is used to optimize the makeup effect of the generated image, and the calculation formula is as follows: In the formula, I0 is the image generated using a pre-trained decoder, and Ψ(·,·) represents the pseudo-truth function, which is the image of the pseudo-makeup dataset made by using histogram matching combined with thin plate spline interpolation. The diffusion reconstruction loss is used to maintain image consistency, aiming to ensure that the generated output remains similar to the source image when the source image is used as a reference image. This diffusion reconstruction loss encourages the improved diffusion model to retain the inherent features and details of the source image, which is beneficial for more stable output and ensures visually consistent results. The calculation formula is as follows: In the formula, This indicates a connection along the spatial dimension between the encoded source image, the encoded source masking image, and the downsampled source masking image; e x,x They are two identical source images connected along the channel dimension; This indicates that the coded reference image, the coded reference occlusion image, and the downsampled reference occlusion image are connected along the spatial dimension; e y,y These are two identical reference images connected along the channel dimension; therefore, the optimization problem of makeup transfer is formulated as follows: In the formula, θ1, θ2, and θ3 represent the training parameters of the makeup repair module, the DINO-V2 pre-trained model, and the facial component perception and matching module, respectively; λ ldm and λ drec These are the weighting coefficients for potential diffusion loss and diffusion reconstruction loss, respectively.

5. The face makeup transfer method based on the improved Diffusion model according to claim 4, characterized in that, In step 3), the specific process of applying the trained optimal makeup transfer model is as follows: First, the face data to be transferred is given, including the corresponding source image and reference image. After data preprocessing, the encoded source image, encoded source occlusion image and downsampled source occlusion image connected along the spatial dimension are obtained and used together as the model input. The output is the makeup transfer result with natural transition.

Images