A realistic makeup transfer method of a three-dimensional avatar capable of animation based on a diffusion model

Abstract

The application provides a realistic makeup transfer method of a three-dimensional movable avatar based on a diffusion model, and specifically comprises the following steps: obtaining a reconstructed three-dimensional drivable target person avatar and a single-view reference photo with a target makeup effect as input of a stable makeup model; generating a series of makeup images by using the stable makeup model, optimizing a global UV map by using the makeup images, and then sampling a makeup picture from the global UV map as supervision information; adding a refinement mechanism to the stable makeup model, and further optimizing makeup details of the target person avatar by using the stable makeup model; and optimizing the makeup of the target person avatar by using a loss function, so as to finally generate a target person avatar with a high-quality makeup effect. The technical scheme of the application overcomes the problems in the prior art that a three-dimensional avatar cannot achieve a high-quality makeup effect and cannot maintain the identity characteristics of the avatar during the makeup process.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of three-dimensional avatar construction, in particular to a realistic makeup transfer method for three-dimensional animatable avatars based on a diffusion model. BACKGROUND

[0002] Three-dimensional avatars are increasingly widely used in virtual reality, gaming, film production, social media and other fields. Three-dimensional avatars not only provide more realistic visual effects, but also enhance user immersion through dynamic expressions and movements. However, in order to meet the needs of users for personalization and aesthetics, customization of three-dimensional avatars has become an important research direction. Among them, the implementation of makeup effects is one of the key factors to improve the visual appeal of three-dimensional avatars.

[0003] Although makeup techniques in two-dimensional images have made significant progress, such as makeup transfer methods implemented through generative adversarial networks (GAN) and stable makeup models, the application of these techniques in three-dimensional avatars still faces many challenges: First, dynamic consistency problem: Existing three-dimensional makeup techniques often have inconsistent makeup effects when dealing with dynamic expressions and multiple perspectives. For example, when the avatar makes exaggerated expressions or is viewed from different angles, makeup details may appear distorted, blurred or lost. This inconsistency seriously affects the realism and user experience of three-dimensional avatars. Then, lack of detail control: In the process of three-dimensional avatar makeup, precise control of makeup details is a key issue. Existing methods are difficult to achieve fine adjustment of makeup details (such as eye shadow, lip gloss, blush, etc.) while maintaining the identity features of the avatar. This results in makeup effects that often lack naturalness and artistic sense. Again, identity preservation problem: The purpose of makeup transfer is to change the appearance of the avatar while preserving its original identity features. However, existing three-dimensional makeup techniques perform poorly in this regard. For example, some methods may change the facial contours, skin color or facial feature of the avatar, resulting in a deviation in identity between the generated avatar and the original avatar. Finally, real-time and efficiency problem: Existing three-dimensional makeup techniques mostly rely on complex optimization processes, making it difficult to achieve real-time rendering. This limits their use in real-time interactive applications such as virtual reality and gaming. For example, some methods based on neural radiance fields (NeRF) can generate high-quality three-dimensional effects, but require a large amount of computing resources and time in the optimization process.

[0004] Therefore, it is necessary to develop a realistic makeup transfer method for three-dimensional animatable avatars that can achieve high-quality makeup effects and preserve the identity features of the avatar during the makeup process. SUMMARY

[0005] The main purpose of the present application is to provide a realistic makeup transfer method for a three-dimensional animatable avatar based on a diffusion model, so as to solve the problem that the three-dimensional avatar cannot achieve high-quality makeup effect in the prior art, and the identity characteristics of the avatar cannot be maintained during the makeup process.

[0006] To achieve the above-mentioned purpose, the present application provides a realistic makeup transfer method for a three-dimensional animatable avatar based on a diffusion model, which specifically comprises the following steps:

[0007] S1, obtaining a reconstructed three-dimensional drivable target character avatar and a single-view reference photo with target makeup effect as the input of a stable makeup model.

[0008] S2, generating a series of makeup images by using the stable makeup model, optimizing the global UV map by using the makeup images, and then sampling the makeup pictures from the global UV map as supervision information, so as to optimize the makeup of the target character avatar.

[0009] S3, adding a refinement mechanism to the stable makeup model, and further optimizing the makeup details of the target character avatar based on the coarse makeup in step S2.

[0010] S4, optimizing the makeup of the target character avatar by combining the loss function, and finally generating a target character avatar with high-quality makeup effect.

[0011] Further, step S1 specifically comprises the following steps:

[0012] S1.1, using existing three-dimensional avatar reconstruction technology, such as Gaussian character model, to generate a three-dimensional drivable target character avatar.

[0013] S1.2, the user provides a single reference picture R with target makeup effect.

[0014] Further, step S2 specifically comprises the following steps:

[0015] S2.1, given k camera perspectives {π1, π2,..., πk}, rendering the corresponding pictures {x1, x2,..., xk} under the target character avatar. k k

[0016] S2.2, adding t time steps of noise to the rendered image x i Then, by using the stable makeup model θ, the denoising process is performed on the noisy picture to generate the makeup transfer images under the k camera perspectives {π1, π2,..., πk}. k

[0017] ​​​​

[0018] S2.3, rendering the mapping image of each point in the makeup transfer image to the UV coordinate by the mesh renderer By accumulating the makeup transfer images of multiple perspectives, the global UV map is optimized, and the optimization formula of the UV map is as follows:

[0019]

[0020] wherein, (H,W), (h,w) are pixel coordinates, is the pixel value at (H,W), and UVmap(h,w) is the UV map pixel value.

[0021] S2.4, using the global UV map, rendering the makeup picture of the target person's avatar in any expression and perspective by the mesh renderer, using step S4 and taking the makeup picture as a supervised picture, optimizing the makeup of the target person's avatar.

[0022] Further, step S3 specifically includes the following steps:

[0023] S3.1, given the target person's avatar obtained by step S2, rendering a series of pictures {y1,y2,...,y n} for the target person's avatar.

[0024] S3.2, selecting the time step t n ∈[20,350] when adding noise to {y1,y2,...,y ′ , obtaining

[0025] S3.3, inputting into the stable makeup model for denoising, and synchronously step S2.2, obtaining the denoised picture

[0026] S3.4, taking as a supervised picture to optimize the target person's avatar, and using step S4, obtaining the optimized makeup of the target person's avatar.

[0027] Further, step S4 specifically includes the following steps:

[0028] S4.1, for the supervised pictures in the two stages of step S2 and step S3, the regions that need to be transferred are optimized by using L1 loss and L LPIPS loss:

[0029] L makeup =L1(M⊙I G ,M⊙I r )+LLPIPS (M⊙I G ,M⊙I r );

[0030] Wherein, M represents the area needing to migrate makeup, I G represents a supervision picture, I r represents a rendering picture, and is an element multiplication.

[0031] S4.2, for the area not needing to migrate makeup, L1 loss is adopted to keep the same texture features as the original character:

[0032] L Res = L1((1-M)⊙I ID ,(1-M)⊙I r );

[0033] Wherein, I ID represents a rendering picture of the initially reconstructed target person portrait.

[0034] The present application has the following beneficial effects:

[0035] The present application realizes high-quality three-dimensional portrait makeup effect by combining a stable makeup model and optimization technology. The consistency of the makeup effect is maintained under dynamic expressions and multiple perspectives, solving the problem in the prior art. By adding a refinement mechanism to the stable makeup model, accurate control of makeup details is realized, improving the naturalness and aesthetics of the makeup effect. The present application maintains the identity features of the portrait during the makeup process, avoiding the destruction of the original features of the portrait by the makeup effect. BRIEF DESCRIPTION OF DRAWINGS

[0036] In order to more clearly illustrate the technical solutions in the specific embodiments or the prior art, the following will briefly introduce the drawings needed to be used in the specific embodiments or the prior art description. Obviously, the drawings described below are some embodiments of the present application, and those skilled in the art can obtain other drawings according to these drawings without creative labor. In the drawings:

[0037] Figure 1 A flowchart of a realistic makeup transfer method of a three-dimensional animatable portrait based on a diffusion model is shown. DETAILED DESCRIPTION

[0038] The technical solutions of the present application will be described below in conjunction with the drawings, obviously, the described embodiments are some of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0039] As Figure 1 shown in a three-dimensional movable avatar based on a diffusion model Realistic makeup transfer method of animation, specifically comprising the following steps:

[0040] S1, obtain a reconstructed three-dimensional drivable target person avatar and a single-view reference photo with target makeup effect as the input of the stable makeup model.

[0041] S2, rough makeup stage: generate a series of makeup images using the stable makeup model, optimize the global UV map using the makeup images, and then sample the makeup pictures from the global UV map as supervision information to optimize the target person avatar makeup.

[0042] S3, fine makeup stage: add a refinement mechanism to the stable makeup model, and on the basis of the rough makeup in step S2, further optimize the makeup details of the target person avatar in the stable makeup model. Further enhance the details of the makeup effect while maintaining consistency with the rough makeup stage.

[0043] S4, optimize the target person avatar makeup by combining the loss function, and finally generate a target person avatar with high-quality makeup effect. By optimizing the features and opacity attributes of the 3D Gaussian distribution, combining the loss function (including L1 loss, L LPIPS loss and limit loss), the avatar is optimized, and finally a three-dimensional avatar with high-quality makeup effect is generated.

[0044] Specifically, step S1 specifically includes the following steps:

[0045] S1.1, generate a three-dimensional drivable target person avatar using existing three-dimensional avatar reconstruction technology, such as Gaussian character model. "Driveable" means that it can change expressions.

[0046] S1.2, the user provides a single reference picture R with target makeup effect.

[0047] Specifically, step S2 specifically includes the following steps:

[0048] S2.1, given k camera perspectives {π1, π2,..., π k}, render the corresponding pictures {x1, x2,..., x k} under the target person avatar. Select 16 camera perspectives {π1, π2,..., π 16} to cover the entire face, and use the Gaussian renderer to render the corresponding pictures {x1, x2,..., x 16} under the target person avatar.

[0049] S2.2, perform a diffusion process. Add noise to the rendered image x i for t time steps to obtain Then, by using the stable makeup model θ, perform a denoising process on the noisy picture to generate k makeup transfer images under camera perspectives {π1, π2,..., π l}.

[0050]

[0051] Add noise to the rendered image x i for y = 1000 time steps to obtain Then, by using the pre-trained stable makeup model θ (Stable-Makeup) based on the diffusion model, perform a denoising process on the noisy picture, and the denoising process uses the denoising diffusion implicit model DDIM to iteratively denoise. Finally, generate makeup transfer images under 16 camera perspectives

[0052] S2.3, render a mapping image of each point in the makeup transfer image to UV coordinates by using a mesh renderer Optimize the global UV map by accumulating makeup transfer images of multiple perspectives, and the optimization formula of the UV map is as follows:

[0053]

[0054] wherein, (H, W) and (h, w) are pixel coordinates, is the pixel value at (H, W), and UVmap(h, w) is the pixel value of the UV map.

[0055] Optimize the global UV map by accumulating makeup images of 16 perspectives to ensure its consistency under different perspectives and expressions. First, render a mapping image of pixels to UV coordinates by using a mesh renderer (mesh renderer), specifically Nvdiffrast.

[0056] S2.4, use the global UV map to render makeup pictures of the target person's avatar under any expression and perspective by using a mesh renderer, and use step S4 and the makeup pictures as supervised pictures to optimize the makeup of the target person's avatar. The pictures rendered when optimizing the makeup of the target person's avatar are used as the rendered pictures of this step.

[0057] Specifically, step S3 specifically includes the following steps:

[0058] S3.1, given the target person's avatar obtained by step S2, render a series of pictures {y1, y2,..., y n} as the rendered picture of this step. At this time, the target character avatar has a rough but consistent makeup in each angle and expression. A series of pictures with different angles and expressions are rendered for the target character avatar.

[0059] S3.2, in order to further improve the quality of makeup, a refinement mechanism is added to the stable makeup model, that is, a noise is added to {y1, y2,..., y n The time step t is selected when the noise is added to ′ ∈[20, 350], and the noise-added picture is obtained

[0060] S3.3, the picture is input into the denoising of the stable makeup model, and the denoised picture is obtained synchronously in step S2.2.

[0061] S3.4, the picture is used as a supervised picture to optimize the target character avatar, and step S4 is used to obtain a high-texture-quality makeup.

[0062] Specifically, step S4 specifically includes the following steps:

[0063] S4.1, for the supervised pictures in the two stages of steps S2 and S3, the regions that need to be transferred are optimized by using L1 loss and L lPIPS loss:

[0064] L makeup =L1(M⊙I G ,M⊙I r )+L LPIPS (M⊙I G ,M⊙I r );

[0065] Wherein, M is a mask obtained by face parsing, representing the region that needs to be transferred, I G represents the supervised picture, I r represents the rendered picture, and ⊙ is element multiplication.

[0066] S4.2, for the regions that do not need makeup transfer, L1 loss is used to keep the same texture features as the original character:

[0067] L Res =L1((1-M)⊙I ID ,(1-M)⊙I r );

[0068] Wherein, I ID represents the rendered picture of the initially reconstructed target character avatar.

[0069] The application is implemented under a PyTorch framework, the model is optimized using an Adam algorithm, and the maximum number of iterations is 12000. Among them, the rough makeup stage is 10000 steps, and the fine stage is 2000 steps.

[0070] In order to verify the feasibility and beneficial effects of the application, the following comparative experiments were carried out. Numerical experiments were carried out on the data set LADN.

[0071] The makeup transfer effect of the current CLIPFace model for driving 3D avatar editing was selected for comparison, and the comparison results are shown in Tables 1 and 2. The CLIPFace model selects mesh as the 3D representation, and uses CLIP loss for makeup transfer.

[0072] The application selects CLIP similarity and DINO similarity as two evaluation indexes to evaluate the trained model. CLIP similarity is used to evaluate the semantic similarity between the final makeup image and the reference image, and the larger the value represents the higher the model accuracy. DINO similarity is used to evaluate the similarity between the final makeup image and the reference image. In order to evaluate the makeup transfer quality of the rendered image under different angles and the quality of the makeup when driving the face, two groups of experiments were carried out. The first group tests the makeup transfer quality under different angles under a single expression. The images are rendered under three horizontal angles: 0°, 45°-45°, and the CLIP similarity and DINO similarity of each group are calculated according to the corresponding angle, and finally the three values are averaged. The comparison results are shown in Table 1. For the second group of experiments, the face is driven to make different expressions under the same angle and the images are rendered, and multi-angle pictures are added to test the CLIP similarity and DINO similarity of these images. The comparison results are shown in Table 2.

[0073] Table 1 Comparison of static multi-angle generation results

[0074]

[0075]

[0076] Table 2 Comparison of dynamic multi-angle generation results

[0077] Model CLIP similarity DINO similarity AvatarMakeup 0.632 0.627 CLIPFace 0.664 0.337

[0078] As can be seen from Table 1 and Table 2, the CLIP similarity obtained using the CLIPFace model is higher in numerical value. This is because the CLIPFace model itself adopts the CLIP loss, the purpose of which is to improve the CLIP similarity, so it will be higher in numerical value, but the numerical value of the DINO similarity is much weaker than the present application. The AvatarMakeup model of the present application has higher numerical values in both CLIP similarity and DINO similarity, which shows the superiority of the present application.

[0079] Of course, the above description is not a limitation on the present application, and the present application is not limited to the above examples. Changes, modifications, additions or substitutions made by those skilled in the art within the spirit and scope of the present application should also be within the protection scope of the present application.

Claims

1. A realistic makeup transfer method of a three-dimensional animatable avatar based on a diffusion model, characterized by, Specifically comprising the following steps: S1, obtaining a reconstructed three-dimensional drivable target character head model and a single reference picture with target makeup effect as an input to the stable makeup model S2, generating a series of makeup images by using a stable makeup model, optimizing the global UV map by using the makeup images, and then sampling the makeup pictures from the global UV map as supervision information to optimize the makeup of the target portrait; S3, adding a refinement mechanism to the stable makeup model, and further optimizing the makeup details of the target portrait on the basis of the coarse makeup in step S2; S4, optimizing the makeup of the target portrait by using a loss function, and finally generating a target portrait with high-quality makeup effect; Step S2 specifically comprises the following steps: S2.1, given camera view rendering a picture in the corresponding view for the target person's avatar ; S2.2, on the rendered image performing time step adding noise to obtain ; then performing makeup stabilization on the image , performing a denoising process on the noisy image to generate a makeup transfer image under the camera view :​ ； S2.3, rendering a mapping image of each point in the makeup transfer image to UV coordinates by a grid renderer , optimizing the global UV map by accumulating makeup transfer images of multiple view angles, and the optimization formula of the UV map is as follows: ； wherein, , 、 is a pixel coordinate, is a pixel value at , is a UV map pixel value; S2.4, using the global UV map to render the makeup pictures of the target portrait under any expression and perspective by using a mesh renderer, and using step S4 to optimize the makeup of the target portrait by using the makeup pictures as supervision pictures; Step S4 specifically comprises the following steps: S4.1, For the two stages of supervision pictures in step S2 and step S3, the area needing to perform makeup transfer is all adopted loss and loss optimization: ； wherein, representing an area where makeup migration needs to be performed, representing a supervision picture, representing a rendered picture, is an element-wise multiplication; S4.2, for the areas where makeup transfer is not desired, use Loss holds the same texture features as the original character: ； wherein, a rendered picture representing the initially reconstructed target person avatar.

2. The realistic makeup transfer method of a three-dimensional animatable avatar based on a diffusion model according to claim 1, wherein, Step S1 specifically comprises the following steps: S1.1, using existing three-dimensional portrait reconstruction technology, such as Gaussian character model, to generate a three-dimensional drivable target portrait; S1.2, the user provides a single reference picture with a target makeup effect . 3.The method of claim 1, wherein, Step S3 specifically comprises the following steps: S3.1, given the target person portrait obtained through step S2, rendering a series of pictures on the target person portrait ; S3.2, to Time step chosen when adding noise , resulting in ; S3.3, to Input denoising in the stabilized makeup model, as in step S2.2, to obtain a denoised picture ; S3.4, obtaining As the supervision picture optimizes the target person's head portrait, and by using step S4, the makeup of the optimized target person's head portrait is obtained.

Images