Method for generating 360-degree face head parameterization model based on synthetic data

Abstract

The application discloses a 360-degree face head parameterization model generation method based on synthetic data, comprising the following steps: face alignment of diversified digital people is carried out by using Procrustes transformation, synthetic 3D face data sets with expression labels are shot and created; a 3DMM model with consistent topology is generated by using a principal component analysis method and Procrustes transformation, and model coefficients are obtained; a parameterized three-dimensional head model capable of generating multi-view multi-expression images is constructed and trained in combination with 3DMM coefficients and expression grid models; the model is fine-tuned to realize separation of hair and face, and image editing is carried out; expression coefficients in a video are extracted, and a digital head with dynamic expressions is generated through the model. The method improves the authenticity and expressiveness of head image generation, and enhances the capability of the model in generating dynamic expressions and multi-view images.

Description

Technical Field

[0001] This invention belongs to the field of computer vision, specifically relating to a method for generating a 360-degree renderable parametric face head model based on synthetic data. Background Technology

[0002] 3D head modeling has long been a popular and ongoing research topic in computer vision and graphics, crucial for many human-related downstream tasks. Generating high-fidelity head models is a highly complex process involving 3D surface reconstruction, material modeling, hairstyle design, and rigging. To address these challenges, traditional workflows rely on expensive systems (such as lighting setups) and unavoidable large amounts of human effort (such as hairstyle design and rigging correction) to generate high-fidelity head models. In recent years, the emergence of neural rendering and neural texturing technologies has made it possible to implement end-to-end workflows using complex neural networks, thereby achieving full automation through data-driven approaches.

[0003] In general, head modeling methods based on neural rendering can be divided into two technical approaches. The first approach utilizes a large set of 2D facial images for training and achieves highly realistic rendering, motion reversal, and identity inversion (Eric RChan, Connor Z Lin, Matthew A Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas J Guibas, Jonathan Tremblay, Sameh Khamis, et al. Efficient geometry-aware 3d generative adversarial networks. In CVPR, pages 16123–16133, 2022. 1, 3, 4, 5, 6). However, its drawbacks are also obvious: it can only achieve high-quality rendering within a small frontal angle range (<±30°), while larger poses will lead to severe image quality degradation. The second approach involves generating 3D models from scanned real faces (Lizhen Wang, Zhiyuan Chen, Tao Yu, Chenguang Ma, Liang Li, and Yebin Liu. Faceverse: a fine-grained and detail-controllable 3D face morphable model from a hybrid dataset. In CVPR, pages 20333–20342, 2022.1, 3) or creating artist-designed models (Erroll Wood, (Aitis, Charlie Hewitt, Sebastian Dziadzio, Thomas J. Cashman, and Jamie Shotton. Fake it till you make it: face analysis in the wildusing synthetic data alone. In CVPR, pages 3681–3691, 2021.1, 33) Learning neural heads. These methods can synthesize high-quality 360-degree renderings; however, their performance in motion animation and identity inversion is poor due to the limited number of 3D models available for training.

[0004] In summary, existing parametric 3D heads are inevitably limited by renderable angles (those trained on a large number of 2D images) or poor binding / identity inversion (those trained on a limited number of 3D head models). RODIN (Tengfei Wang, Bo Zhang, Ting Zhang, Shuyang Gu, Jianmin Bao, Tadas Baltrusaitis, Jingjing Shen, Dong Chen, Fang Wen, Qifeng Chen, et al. Rodin: A generative model for sculpting 3D digital avatars using diffusion. In CVPR, pages 4563–4573, 2023.1,3) trains a parametric head using large-scale multi-view images rendered from 3D models designed by artists, while PanoHead (Sizhe An, Hongyi Xu, Yichun Shi, Guoxian Song, Umit Y Ogras, and Linjie Luo. Panohead: Geometry-aware 3D full-head synthesis in 360deg. In CVPR, pages 4563–4573, 2023.1,3) trains a parametric head using large-scale multi-view images rendered from 3D models designed by artists. The model (20950–20959, 2023.3,5,6) was trained using a combination of large-scale frontal 2D images and captured rear-view hairstyle images, both of which resulted in 360-degree renderable parametric head models. A major problem with these two models is that their facial expressions and movements cannot be bound together. Summary of the Invention

[0005] To generate detailed 3D face models directly from text, this invention provides a method for generating high-fidelity, 360-degree renderable parametric 3D head models based on synthetic data.

[0006] To achieve the above-mentioned objectives, the technical solution adopted by this invention is as follows:

[0007] A method for generating a 360-degree parametric model of a human face head based on synthetic data includes the following steps:

[0008] S1. Acquire several digital humans with different skin tones, hairstyles, and appearances, and align their facial regions according to the Prototype transform; use an imaging device to photograph the aligned digital humans, select an expression base as the basic expression, and create a high-fidelity synthetic 3D face dataset with expression labels.

[0009] S2, Based on the synthetic 3D face dataset, obtain topologically consistent mesh models for all digital human expressions, i.e., models with the same number of vertices and faces; apply the scaling transformation, rotation matrix, and translation matrix corresponding to the Prototype transform to the mesh model to ensure topological consistency while maintaining alignment in three-dimensional space; use principal component analysis to reduce the dimension of the aligned mesh model with expressions to generate a 3DMM model and obtain the corresponding 3DMM coefficients;

[0010] S3, the synthesized 3D face dataset is used as a one-hot encoding for the expression dimension, and the 3DMM coefficients are used as encoding for the shape dimension. Combined with the spatially aligned mesh model with expressions, training data is constructed to generate a three-dimensional feature plane and a learnable neural texture encoding; a parameterized 3D head model is constructed and trained, which is used to generate multi-view, multi-expression head images given the 3DMM coefficients.

[0011] S4: Construct several datasets of digital humans without hair, and continue to fine-tune the parametric 3D head model obtained in training step S3, separating the hair from the face; use a text-based image conditional diffusion model to edit the head in the parametric 3D head model;

[0012] S5. Take a series of videos, extract the facial expression coefficient parameter stream based on the expression base in the videos, obtain the digital human and the mesh model under the corresponding expression in the video according to the 3DMM model and the facial expression coefficient, select texture encoding, and send it into the parameterized 3D human head model trained in step S4 to generate a digital human with dynamic expression that can be rendered in 360 degrees.

[0013] This invention proposes a method for generating high-fidelity, 360-degree renderable parametric 3D human head models based on synthetic data. To this end, a high-fidelity human head dataset is constructed, and a novel framework is established to learn from the synthetic dataset. This invention separates facial motion, facial shape, and facial appearance, representing them respectively using classic hybrid shape models and neural textures, achieving decoupling of the generated facial shape, appearance, and expression. Unlike all existing methods, this invention is the first to generate parametric 3D human head models that support 360-degree free-viewpoint synthesis, single-image fitting, and animation driven by a standard hybrid shape parameter flow. This method can be widely applied in fields such as digital humans, game creation, and film special effects, possessing high practical value and promising development prospects. Attached Figure Description

[0014] Figure 1 This is a flowchart of the method of the present invention.

[0015] Figure 2 This is a schematic diagram of fitting a human face in an embodiment of the present invention.

[0016] Figure 3 This is a flowchart of the operation phase in an embodiment of the present invention.

[0017] Figure 4 This is a diagram illustrating the hairstyle change and driving results in an embodiment of the present invention.

[0018] Figure 5 This is a diagram illustrating the face editing and driving results in an embodiment of the present invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] like Figure 1 As shown, this invention proposes a method for generating a high-fidelity, 360-degree renderable parametric 3D human head model based on synthetic data. This method is used for 360-degree free-viewpoint synthesis and enables single-image fitting, animation production, and text-based editing. The method includes the following steps:

[0021] S1. Several digital humans with different skin tones, hairstyles, and appearances are created using software such as Adobe Fuse or Unreal Engine, and their facial regions are aligned using Prototype transform. In this embodiment, the aligned digital humans are captured by a camera from 24 horizontal perspectives and 3 vertical perspectives, for a total of 72 perspectives. Simultaneously, 52 expression bases proposed by ARKit are used as basic expressions to create a high-fidelity synthetic 3D face dataset with expression labels.

[0022] S2, based on a synthetic 3D face dataset with facial expression labels, exports mesh models of all digital humans for each facial expression from the software and processes them to be topologically consistent, i.e., models with the same number of vertices and faces; the scaling transformation, rotation matrix, and translation matrix corresponding to the Protodyakonov transformation are used in the mesh model to ensure that they are topologically consistent while maintaining alignment in 3D space; principal component analysis is used to reduce the dimensionality of the aligned mesh models with facial expressions to generate a 3DMM model, and the corresponding 3DMM coefficients are obtained.

[0023] S3, using the face dataset from step S1 as a one-hot encoding for the expression dimension, and the 3DMM coefficients generated from step S2 as the encoding for the shape dimension, combined with the spatially aligned mesh model with expressions, to construct training data, generating three feature planes and learnable neural texture encoding; constructing and training a parameterized 3D head model, which is used to generate multi-view, multi-expression head images given the 3DMM coefficients. During training, the images generated after rendering through the three feature planes are compared with the images in the dataset using a loss function calculation to optimize the overall model network parameters.

[0024] S4. Based on the parameterized 3D head model trained in step S3, following the method of creating a 3D face dataset in step S1, construct a small amount of data of 100 bald digital humans, and continue to optimize and train the parameterized 3D head model. By separating the hair from the face, the effect of changing the hairstyle can be achieved by replacing the three-dimensional feature planes related to the hairstyle, that is, replacing the original feature planes of the hairstyle with another set of feature planes representing the hairstyle; at the same time, the head in the model is edited using the existing text-based image conditional diffusion model.

[0025] S5. Shoot a series of videos, extract the facial expression coefficient parameter stream based on ARKit from the video, obtain the digital human and the 3D mesh model of the corresponding expression in the video according to the 3DMM model and the expression coefficient, select texture encoding, and feed it into the parameterized 3D human head model trained in step S4 to generate a digital human with dynamic expression that can be rendered in 360 degrees.

[0026] The specific implementation steps of the above solution are as follows:

[0027] 1. A high-fidelity synthetic 3D face dataset was created for use in this embodiment, containing 100 different digital humans, 50 males and 50 females, with diverse hairstyles and facial appearances. Each subject was bound to 52 standard hybrid shapes. These 3D heads were rendered by 72 head-centric virtual cameras, covering 3 pitch angles and 24 horizontal rotation angles, specifically by capturing a frame at 15-degree intervals in the horizontal direction, and 30 degrees overhead, 30 degrees up, and a parallel viewpoint in the vertical direction. The parametric 3D head models in this embodiment contain detailed textures at the pore level, making them more realistic than the data used to train RODIN.

[0028] 2. Parametric 3D Mesh Model. Given a shape code *s* representing facial motion and a mixed shape / expression code *b* related to the expression basis, a 3DMM model can be generated through a parametric 3D mesh model mapping. This mapping is established by performing dimensionality reduction on the identity dimension in the synthetic dataset. Specifically, 3D mesh models of 100 digital humans are extracted based on 52 expressions using ARKit (Apple.Arkit.https: / / developer.apple.com / augmented-reality / arkit / , 2023.5), with each 3D mesh consisting of 2548 vertices and 5048 faces. A matrix is ​​created. Where N is the number of fixed points. Principal Component Analysis (PCA) is used to extract a smaller core tensor from the identity dimension. Therefore, the mapping relationship can be summarized by the following formula:

[0029] V = C r ×s×b

[0030] The resulting V is an N×3 vertices of a digital head mesh model. The vertex coordinates of the mesh model are used as input to the model, and the model outputs the corresponding 3DMM coefficients.

[0031] 3. The shape code s corresponding to each character is processed by a mapping network to generate a texture vector, and then passed through a fully connected network to obtain the neural texture (Thies, J., M., M.:Deferred neuralrendering:Image synthesis using neural textures.ToG 38(4),1-12(2019)6), storing each person's appearance texture information in high-dimensional features. That is, each person has a corresponding neural texture encoding. Fixing the learned neural texture encoding t, the neural texture and feature map of the three-dimensional feature plane are synthesized through the image generation network. Following Next3D (Jingxiang Sun, Xuan Wang, Lizhen Wang, Xiaoyu Li, YongZhang, Hongwen Zhang, and Yebin Liu.Next3d:Generative neural texture rasterization for 3d-aware headavatars.InCVPR,pages 20991–21002,2023.1,3,4,5), the neural texture with the generated 3D mesh model is projected onto the three-dimensional feature plane, and then rendered into an image according to the camera parameters. Finally, RefineNet is used to further improve the details of the rendered image. Based on the representation of NeRF (Mildenhall, B., Srinivasan, PP, Tancik, M., Barron, JT, Ramamoorthi, R., Ng, R.: Nerf: Representing scenes as neural radiance fields for view synthesis. In: ECCV.pp.405-421(2020)4,6) based on neural textures and 3D mesh-conditional three-dimensional feature planes, this embodiment further proposes three modifications. First, in constructing the three-dimensional feature planes, unlike the existing method of using three feature planes to be rasterized, this embodiment uses six feature planes including front, back, left, right, top, and bottom to simulate the 360-degree appearance of the head. The 3D mesh model is rasterized and projected in six directions (front, back, left, right, top, and bottom), and the planes in the front, back, left, right, and top directions are connected to form a new three-plane representation, which is the process of the rasterization module. Through experiments, it was observed that this strategy improves the rendering quality and is more suitable for training on 360-degree multi-view image data. Secondly, a dual-branch network is introduced to model the bald model and the model with hair separately. This invention proposes a training strategy that separates hair and head to allow for free change of hairstyles and better fitting performance.The training method involves rendering a small number of multi-view images of all digital humans without hair. Based on the labels corresponding to hairstyles in the real images, the method of combining the hairstyle-related three-dimensional feature planes with the face-related three-dimensional feature planes is determined. Network parameters are fine-tuned while keeping the trainer constant. Ultimately, the hairstyle-related three-dimensional feature planes retain hairstyle-related information, allowing the network model to generate bald and hairstyle-related face images based on hairstyle labels. The hairstyle is replaced by replacing the hairstyle-related three-dimensional feature planes. Finally, a more precise parametric 3D mesh model is used, which is connected to the rasterization module. Although the parametric 3D mesh model is not trained together with the neural texture, this connection makes shape encodings an optimizable parameter in GAN inversion, thereby improving the performance of single-image fitting tasks.

[0032] 4. The loss function used to train the triaxial feature plane of the neural texture and parameterized 3D mesh model is constructed as follows:

[0033]

[0034] in Let be the photometric loss function. It is the dual GAN ​​loss proposed in Next3D; This is the density regularization term proposed by Eric in EG3D. λ1 and λ2 are the weights corresponding to each loss function. In this invention, the weight λ1 is 0.01 and follows the settings of EG3D. The generative adversarial network includes a generator and a discriminator, and the learning rate l is related to the regularization interval R of the generator and discriminator. i Related:

[0035]

[0036] The generator R in this invention i The value is 4, and the R value of the discriminator is 4. i The base learning rate is 16; the model is trained using the Adam optimizer with a base learning rate of l. base The value was 0.0025, and the batch size was 8. The training time on 4 NVIDIA 3090 GPUs was 10 days.

[0037] A dual three-dimensional feature plane is generated for both hairless and hair-containing head models to achieve head and hair separation. First, a full network with two learnable three-dimensional feature planes is trained for 3000k iterations using images with hair. Then, the model is fine-tuned for an additional 40k iterations to achieve hair separation. During fine-tuning, only the hair-related three-dimensional feature planes are learnable. The optimizer's hyperparameters remain unchanged throughout both training phases.

[0038] Conditional GANs are introduced to further enhance the details of generated faces. Following the approach of MoFaNeRF, pix2pixHD (Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. High-resolution image synthesis and semantic manipulation with conditional gans. In CVPR, pages 8798–8807, 2018.4) is used as the backbone network of RefineNet. The input to RefineNet is the generated image rendered from the synthesized three-dimensional feature plane. During the training phase, the entire network except RefineNet is trained first, and then RefineNet is trained separately with the other network parts trained separately. The loss function for training RefineNet is:

[0039]

[0040] in These are the generative adversarial network loss, structural similarity loss, and perceptual image quality assessment loss, respectively. μ1 and μ2 are the loss weights corresponding to each loss function, following the settings of pix2pixHD.

[0041] 5. After step 4, a parametric 3D head model capable of supporting 360-degree free-view rendering can be obtained. A series of videos are captured, and the facial expression coefficient parameter stream based on the stated expression base is extracted. Based on the 3DMM model and the expression coefficients, a mesh model of the digital human and the corresponding expression in the video is obtained. Texture encoding is selected and fed into the parametric 3D head model to generate a digital human with dynamic expressions that can be rendered in 360 degrees. The result is as follows: Figure 4 As shown.

[0042] This embodiment also employs the model proposed in this invention to complete the task of fitting a single image. The main idea is to keep the model's training parameters unchanged and update the latent input space according to the gradient, similar to GAN inversion. However, several modifications were made to the model to improve fitting performance. The main challenge lies in the fact that the obtained parameterized 3D head model was trained on a relatively small dataset of only 100 people, thus the parameter space of the texture encoding t is very limited and difficult to cover diverse facial appearances. To address this problem, neural textures are optimized instead of latent or mapping encodings w, thereby improving the generalization ability of the fitting. The patch mesh is also optimized through a linear mapping from s to the 3D mesh and by connecting them to the rasterization module.

[0043] In the single-image fitting task, the L1 loss between the generated frontal view and the input image is used as a constraint. First, shape encodings are optimized to obtain a fitted patch mesh that matches the input image. Then, the fitted shape is fixed, and a randomly selected neural texture is used as initialization to optimize the neural texture. The image is first normalized, aligning the face of the input image with the facial region of the 3DMM model. The image generated by the 3D mesh model from the patch mesh initialized with the random texture is then fused with a Poisson blend to even out skin color using the aligned real-world image, serving as the preprocessed real image. By optimizing the triaxial feature plane space to reduce the difference between the preprocessed image and the image generated by the random texture, the neural texture and the latent encoding of the hair can be optimized, thereby generating a 360-degree renderable and animable head that fits the appearance of the input image.

[0044] The 3DMM model obtained in this invention is defined based on the standard 52 facial expression bases, which are used by ARKit. Therefore, the generated or fitted head can be generated by obtaining parameter stream data through a series of facial expression base media tools. Simultaneously, it can generate virtual existence phenomena of non-tightly attached, permanently consistent appearance features such as hair, mouth, and eyelashes. Therefore, this invention demonstrates that the proposed learnable representation method can achieve the performance of complex head models designed by traditional artists, achieving high-fidelity animation effects.

[0045] This invention combines a text-based image conditional diffusion model (Tim Brooks, Aleksander Holynski, and Alexei A. Efros. Instructpix2pix: Learning to follow image editing instructions. In CVPR, pages 18392–18402, 2023.5) with this invention for 3D head editing. Specifically, after generating or fitting a head, multi-view images are rendered and edited using a text-guided image-to-image converter. The edited images are then used for fine-tuning the parametric 3D head model. During fine-tuning, the optimizer and learning rate remain unchanged compared to before. Experience shows that when fine-tuning the parametric model, sampling on the generated images using a patch approach yields better results than the original random sampling strategy. The text-based editing strategy in this invention aims to add abstract features, such as "makeup" and "aging," to the generated head without altering the person's identity or appearance. The edited head can still be animated and driven using standard facial expression parameter streams.

[0046] Through the above steps, this invention can directly generate a high-fidelity, driven 3D face model from a frontal photograph or a video description, and change its hairstyle. The generated result is as follows: Figure 5 As shown, even after changing the digital human's texture or hairstyle, it still maintains the same video-based expression-driven effect.

[0047] This invention proposes a high-fidelity, 360-degree renderable parametric 3D head model generation method based on synthetic data. This method directly generates a full 3D head model from parameters, reducing the cost of acquiring parametric 3D models and contributing to research on 3D face generation. The generated 3D faces can be applied in multiple fields such as film, games, and security. This invention is the first method to simultaneously achieve 360-degree free-view synthesis, image-based inversion, and animation in a parametric 3D full-head model. It not only improves the realism and expressiveness of generated head images but also enhances the model's ability to generate dynamic expressions and multi-view images. Experiments show that facial motion and appearance are well separated in the parameter space, achieving state-of-the-art performance in rendering and animation quality.

Claims

1. A method for generating a 360-degree face head parametric model based on synthetic data, characterized in that, Comprise the following steps: S1, obtain a plurality of digital people with different skin colors, hair styles and appearances, and align their facial regions according to the Procrustes transformation; use an imaging device to take pictures of the aligned digital people, select an expression base as the basic expression, and create a high-fidelity synthetic 3D face dataset with expression labels; S2, based on the synthetic 3D face dataset, obtain the topologically consistent mesh models of all digital people under each expression, that is, models with the same number of vertices and faces; use the scale transformation, rotation matrix and translation matrix corresponding to the Procrustes transformation for the mesh model, so that they are topologically consistent while ensuring the alignment relationship in three-dimensional space; use principal component analysis to reduce the dimension of the aligned mesh model with expression to generate a 3DMM model, and obtain the corresponding 3DMM coefficient; S3, use the synthetic 3D face dataset as the one-hot encoding of the expression dimension, use the 3DMM coefficient as the encoding of the shape dimension, combine the spatially aligned mesh model with expression, construct training data, generate a three-way feature plane and a learnable neural texture code; construct and train a parameterized three-dimensional head model, which is used to generate multi-view and multi-expression head images under the condition of known 3DMM coefficient; S4, construct a plurality of digital people data without hair, continue to fine-tune the parameterized three-dimensional head model obtained in step S3 to separate the hair from the face; use a text-based image conditional diffusion model to edit the head in the parameterized three-dimensional head model; S5, shoot a group of videos, extract the expression coefficient parameter stream of the face in the video based on the expression base, obtain the mesh model of the digital person and the corresponding expression in the video based on the 3DMM model and the expression coefficient, select the texture code, and input it into the parameterized three-dimensional head model trained in step S4 to generate a digital person with dynamic expression that can be rendered by 360 degrees.

2. The method of claim 1, wherein, In step S1, when using an imaging device to take pictures of the aligned digital people, fix the position of the digital people in the shooting space, fix the distance between the imaging device and the digital people, and take multi-view pictures at an interval of 15 degrees in the horizontal direction and at a downward view angle of 30 degrees, an upward view angle of 30 degrees and a parallel view angle. 3.The method of claim 1, wherein, In step S2, the number of vertices of each mesh model is 2548, and the number of faces is 5048.

4. The method of claim 1, wherein, In step S3, when generating a three-way feature plane, rasterize and project the mesh model in six directions: front, back, left, right, up and down, and connect the planes in the front-back, left-right and up-down directions as a plane to form a new three-plane representation.

5. The method of claim 1, wherein, In step S3, the parameterized three-dimensional head model is trained using an L1 loss function and a weighted generative adversarial network loss: wherein is a luminance loss function, is a dual GAN loss; is a density regularization term, and λ1and λ2are weights corresponding to each loss function.

6. The method of claim 5, wherein the generating the 360-degree face head parameterization model based on the synthetic data comprises: The generative adversarial network includes a generator and a discriminator, a learning rate l and a regularization interval R of the generator and the discriminator i Related: wherein, l base is the base learning rate.

7. The method of claim 6, wherein the generating the 360-degree face head parameterization model based on the synthetic data comprises: In step S4, when fine-tuning the parameterized three-dimensional head model, the loss function, optimizer and learning rate of the network are consistent with those in step S3.

8. The method of claim 1, wherein, In step S4, when fine-tuning the parameterized three-dimensional head model, a refinement network is established, and the loss function of the refinement network is: wherein and are the generative adversarial network loss, structural similarity loss and perceptual image quality assessment loss, respectively; and μ1 and μ2 are the loss weights corresponding to each loss function, respectively.

Images