360-degree face head parameterized model generation method based on synthetic data

Abstract

The invention discloses a 360-degree human face head parameterization model generation method based on synthetic data. The method comprises the following steps: carrying out human face alignment on diversified digital people by utilizing Prschner transformation, shooting and creating a synthetic 3D human face data set with expression labels; generating a topologically consistent 3DMM model by using a principal component analysis method and Princiteur transform, and obtaining a model coefficient; constructing and training a parameterized three-dimensional head model capable of generating a multi-view multi-expression image in combination with the 3DMM coefficient and the expression grid model; fine-tuning the model to realize hair and face separation, and performing image editing; and extracting an expression coefficient in the video, and generating a digital head with a dynamic expression through a model. The method provided by the invention not only improves the authenticity and expressive force of head image generation, but also enhances the ability of the model in the aspect of generating dynamic expressions and multi-view images.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision, and in particular relates to a method for generating a 360-degree renderable parameterized human face head model based on synthetic data. Background Art

[0002] 3D head modeling has been a long-standing and hot research topic in computer vision and graphics, and is crucial to many downstream tasks related to the human body. The process of generating a high-fidelity head model is very complex, involving 3D surface reconstruction, material modeling, hair design, binding, etc. To address these challenges, traditional processes rely on expensive systems (such as lighting settings) and inevitable large amounts of manpower (such as hair design and binding correction) to generate high-fidelity head models. In recent years, the emergence of neural rendering and neural texturing technologies has made it possible to implement end-to-end processes with complex neural networks, thereby achieving full automation through data-driven methods.

[0003] In general, the head modeling methods based on neural rendering can be divided into two technical routes. The first route uses a large set of 2D facial images for training and achieves highly realistic rendering, motion redirection, and identity inversion (Eric RChan, Connor Z Lin, Matthew A Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas JGuibas, Jonathan Tremblay, Sameh Khamis, et al. Efficient geometry-aware 3d generative adversarial networks. In CVPR, pages 16123–16133, 2022.1, 3, 4, 5, 6). But its shortcomings are also obvious, that is, it can only achieve high-quality rendering within a small angle range of the front (<±30°), and larger postures will lead to serious image quality degradation. The second route is to generate 3D models from scanned real faces (Lizhen Wang, Zhiyuan Chen, Tao Yu, Chenguang Ma, Liang Li, and Yebin Liu. Faceverse: a fine-grained and detail-controllable 3d face morphable model from a hybrid dataset. In CVPR, pages 20333–20342, 2022.1, 3) or models designed by artists (Erroll Wood, aitis, Charlie Hewitt, Sebastian Dziadzio, Thomas J Cashman, and Jamie Shotton. Fake it till you make it: face analysis in the wild using synthetic data alone. In CVPR, pages 3681–3691, 2021.1, 33) Learning neural heads. These methods can synthesize high-quality 360-degree renderings, however, due to too few 3D models available for training, the performance of motion animation and identity inversion is poor.

[0004] In summary, existing parametric 3D heads are inevitably limited in renderable angles (those trained on a large number of 2D images) or have poor binding / identity inversion (those trained on a limited number of 3D head models). RODIN (Tengfei Wang, Bo Zhang, Ting Zhang, Shuyang Gu, Jianmin Bao, Tadas Baltrusaitis, Jingjing Shen, Dong Chen, Fang Wen, Qifeng Chen, et al. Rodin: A generative model for sculpting 3D digital avatars using diffusion. In CVPR, pages 4563–4573, 2023.1, 3) trains parametric heads using large-scale multi-view images rendered from artist-designed 3D models, while PanoHead (Sizhe An, Hongyi Xu, Yichun Shi, Guoxian Song, Umit Y Ogras, and Linjie Luo. Panohead: Geometry-aware 3D full-head synthesis in 360deg. In CVPR, pages 20950–20959,2023.3,5,6) combine large-scale frontal 2D images and captured rear-view hairstyle images for training, both of which achieve 360-degree renderable parametric head models. The main problem with these two models is that their facial expressions and movements cannot be bound. Summary of the invention

[0005] In order to directly generate a fine three-dimensional face model from text, the present invention provides a method for generating a high-fidelity 360-degree renderable parameterized three-dimensional head model based on synthetic data.

[0006] In order to achieve the above-mentioned invention object, the technical solution adopted by the present invention is as follows:

[0007] The method for generating a 360-degree human face head parametric model based on synthetic data comprises the following steps:

[0008] S1, obtain several digital humans with different skin colors, hairstyles and appearances, and align their facial regions according to the Prototype Transform; use imaging equipment to shoot the aligned digital humans, select an expression base as the basic expression, and create a high-fidelity synthetic 3D face dataset with expression labels;

[0009] S2, based on the synthetic 3D face dataset, obtain topologically consistent mesh models of all digital humans under various expressions, that is, models with the same number of vertices and facets; apply the scale transformation, rotation matrix, and translation matrix corresponding to the Prototype transformation to the mesh models, so that they are topologically consistent while ensuring alignment in three-dimensional space; use the principal component analysis method to reduce the dimension of the aligned mesh models with expressions to generate a 3DMM model, and obtain the corresponding 3DMM coefficients;

[0010] S3, using the synthesized 3D face dataset as a one-hot encoding of the expression dimension, using the 3DMM coefficient as the encoding of the shape dimension, combining the spatially aligned mesh model with expressions, constructing training data, generating a three-dimensional feature plane and a learnable neural texture encoding; constructing and training a parameterized three-dimensional head model, which is used to generate multi-view and multi-expression head images when the 3DMM coefficients are known;

[0011] S4, constructing data of several digital humans without hair, continuing to fine-tune the parameterized three-dimensional head model obtained in the training step S3, separating the hair from the face; using the text-based image conditional diffusion model, editing the head in the parameterized three-dimensional head model;

[0012] S5, shoot a group of videos, extract the expression coefficient parameter flow of the face in the video based on the expression base, obtain the mesh model of the digital human and the corresponding expression in the video according to the 3DMM model and the expression coefficient, select the texture coding, and send it into the parameterized three-dimensional human head model trained in step S4 to generate a digital human with dynamic expressions and can be rendered in 360 degrees.

[0013] The present invention proposes a method for generating a high-fidelity 360-degree renderable parametric three-dimensional human head model based on synthetic data. For this purpose, a high-fidelity human head dataset is constructed, and a novel framework is established to learn from the synthetic dataset. The present invention separates facial motion, facial shape and facial appearance, which are represented by classic mixed shape models and neural textures respectively, so as to achieve the effect of decoupling the generated human facial shape, appearance and expression. Different from all the above existing methods, the present invention is the first method that can generate a parametric 3D human head model that supports 360-degree free perspective synthesis, single image fitting and animation driven by a standard mixed shape parameter flow. The method can be widely used in the fields of digital humans, game creation, movie special effects, etc., and has high practical value and development prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 Flow chart of the method of the present invention.

[0015] Figure 2 Schematic diagram of fitting a human face in an embodiment of the present invention.

[0016] Figure 3 Flow chart of the operation phase in an embodiment of the present invention.

[0017] Figure 4 This is a diagram showing the hairstyle change and driving results in an embodiment of the present invention.

[0018] Figure 5 This is a diagram showing the edited face and driving results in an embodiment of the present invention. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0020] like Figure 1 As shown, the present invention proposes a method for generating a high-fidelity 360-degree renderable parametric 3D human head model based on synthetic data, which is used for 360-degree free perspective synthesis and realizes single image fitting, animation production and text-based editing. The method comprises the following steps:

[0021] S1, using software such as Adobe Fuse or Unreal Engine to create several digital humans with different skin colors, hairstyles, and appearances, and aligning their facial regions according to the Prototype Transform. In this embodiment, the camera shoots the aligned digital humans from 24 viewing angles in the horizontal direction and 3 viewing angles in the vertical direction, a total of 72 viewing angles, and uses the 52 expression bases proposed by ARKit as basic expressions to create a high-fidelity synthetic 3D face dataset with expression labels.

[0022] S2, based on the synthetic 3D face dataset with expression labels, exports the mesh models of all digital humans under various expressions from the software and processes them into topologically consistent models, that is, models with the same number of vertices and faces; the scale transformation, rotation matrix, and translation matrix corresponding to the Platts transform are used for the mesh models to ensure that they are topologically consistent while ensuring alignment in three-dimensional space; the principal component analysis method is used to reduce the dimensionality of the aligned mesh models with expressions to generate a 3DMM model, and the corresponding 3DMM coefficients are obtained.

[0023] S3, using the face dataset in step S1 as the one-hot encoding of the expression dimension, using the 3DMM coefficients generated in step S2 as the encoding of the shape dimension, combining with the spatially aligned grid model with expressions, constructing training data, generating three feature planes and learnable neural texture encoding; constructing and training a parameterized three-dimensional head model, which is used to generate multi-view and multi-expression head images when the 3DMM coefficients are known. When training the model, the images generated after the three-dimensional feature planes are rendered are compared with the images in the dataset to calculate the loss function and optimize the overall model network parameters.

[0024] S4, based on the parameterized three-dimensional head model trained in step S3, a small amount of data of 100 digital people without hair is constructed in imitation of the method of creating a 3D face data set in step S1, and the parameterized three-dimensional head model is continuously optimized and trained. By separating the hair from the face, the hairstyle of the person can be replaced by replacing the three-dimensional feature plane related to the hairstyle, that is, replacing the feature plane of the original hairstyle with another set of feature planes representing the hairstyle; at the same time, the head in the model is edited using the existing text-based image conditional diffusion model.

[0025] S5, shoot a group of videos, extract the expression coefficient parameter stream of the face in the video based on ARKit, obtain the 3D mesh model of the digital human with the corresponding expression in the video according to the 3DMM model and the expression coefficient, select the texture encoding, and send it to the parameterized 3D human head model trained in step S4, so as to generate a digital human with dynamic expressions and can be rendered in 360 degrees.

[0026] The specific implementation steps of the above solution are as follows:

[0027] 1. Create a high-fidelity synthetic 3D face dataset for use in this embodiment, including 100 different digital people, 50 of whom are male and 50 of whom are female, with diverse hairstyles and facial appearances, and each subject is bound to 52 standard blended shapes. These 3D heads are rendered by 72 head-centered virtual cameras, covering 3 pitch angles and 24 horizontal rotation angles, specifically taking a frame at intervals of 15 degrees in the horizontal direction, and 30 degrees of looking down, 30 degrees of looking up, and parallel viewing angles in the vertical direction. The parameterized 3D head model of this embodiment contains detailed textures at the pore level, which is more realistic than the data used to train RODIN.

[0028] 2. Parameterized 3D mesh model. Given the shape code s representing facial movement and the mixed shape / expression code b associated with the expression base, a 3DMM model can be generated by mapping the parameterized 3D mesh model. This mapping is established by implementing dimensionality reduction on the identity dimension in the synthetic dataset. Specifically, 3D mesh models of 100 digital humans with 52 expressions based on ARKit (Apple.Arkit.https: / / developer.apple.com / augmented-reality / arkit / ,2023.5) are extracted, and each 3D mesh consists of 2548 vertices and 5048 patches. Create a matrix Where N is the number of fixed points. Use principal component analysis PCA to extract a smaller core tensor in the identity dimension Therefore, the mapping relationship can be summarized by the following formula:

[0029] V=C r ×s×b

[0030] The obtained V is the N×3 vertices of a digital head mesh model. The vertex coordinates of the mesh model are used as the input of the model, and then the model outputs the corresponding 3DMM coefficients.

[0031] 3. The shape code s corresponding to each character will pass through the mapping network to generate a texture vector, and then pass through a network composed of a fully connected layer to obtain a neural texture (Thies, J., M., M.:Deferred neural rendering: Image synthesis using neural textures. ToG 38(4), 1-12(2019)6), the appearance texture information of each person is stored in high-dimensional features. That is, each person has a corresponding neural texture code. The learned neural texture code t is fixed, and the feature map of the neural texture and the three-way feature plane is synthesized through the image generation network. Following Next3D (Jingxiang Sun, Xuan Wang, Lizhen Wang, Xiaoyu Li, YongZhang, Hongwen Zhang, and Yebin Liu. Next3d: Generative neural texture rasterization for 3d-aware headavatars. InCVPR, pages 20991–21002, 2023.1, 3, 4, 5), the neural texture with the generated 3D mesh model is projected onto the three-way feature plane, and then rendered into an image according to the camera parameters. Finally, RefineNet is used to further improve the details of the rendered image. Based on the representation of the three-way feature plane NeRF (Mildenhall, B., Srinivasan, PP, Tancik, M., Barron, JT, Ramamoorthi, R., Ng, R.: Nerf: Representing scenes as neural radiance fields for view synthesis. In: ECCV. pp. 405-421 (2020) 4, 6) of neural texture and 3D grid conditions, this embodiment further proposes three modifications. First, when constructing the three-way feature plane, unlike the existing use of three feature planes to be rasterized, this embodiment uses six feature planes including front, back, left, right, top and bottom to simulate the 360-degree appearance of the head, and rasterizes the 3D grid model to six directions of front, back, left, right, top and bottom, and sets the planes in the front, back, left, right, top and bottom directions to be connected as a plane to form a new three-plane representation, which is the process of the rasterization module. Through experiments, it is observed that this strategy improves the rendering quality and is more suitable for training on 360-degree multi-view image data. Secondly, a two-branch network is introduced to model the bald model and the model with hair separately. The present invention proposes a training strategy to separate the hair and the head to achieve free replacement of hairstyle and better fitting performance.The training method is as follows: render a small number of multi-view images of all digital people without hair, determine the combination of the hairstyle-related three-dimensional feature plane and the face-related three-dimensional feature plane according to whether the real image has a label with a hairstyle, fine-tune the network parameters, ensure that the trainer remains unchanged, and finally achieve the hairstyle-related three-dimensional feature plane to retain the hairstyle-related information, so that the network model can generate bald faces and face images with hairstyles according to the hairstyle labels, and achieve the effect of replacing the hairstyle by replacing the hairstyle-related three-dimensional feature plane. Finally, a more accurate parameterized 3D mesh model is used, which is connected to the rasterization module. Although the parameterized 3D mesh model is not trained with the neural texture, this connection makes the shape encoding s an optimizable parameter in the GAN inversion, thereby improving the performance of the single image fitting task.

[0032] 4. The loss function for training the three-way feature plane based on neural texture and parameterized 3D mesh model is constructed as:

[0033]

[0034] in is the photometric loss function, It is the dual GAN ​​loss proposed in Next3D; It is the density regularization term proposed by Eric in EG3D. 1 and λ 2 is the weight corresponding to each loss function. In the present invention, the weight λ 1 is 0.01 and follows the setting of EG3D. The generative adversarial network includes a generator and a discriminator, and the learning rate l and the regularization interval R of the generator and the discriminator are i Related:

[0035]

[0036] The R of the generator in the present invention i is 4, the discriminator’s R i is 16; the model is trained using the Adam optimizer, with a basic learning rate of l base The value is 0.0025 and the batch size is 8. The training time is 10 days on 4 NVIDIA3090 GPUs.

[0037] Generate dual three-way feature planes for the head model without hair and the head model with hair to achieve the separation effect of head and hair. First, the full network of two learnable three-way feature planes is trained for 3000k iterations using images with hair. Then, the model is fine-tuned for an additional 40k iterations to achieve hair separation. During the fine-tuning process, only the three-way feature planes related to hair are learnable. During these two training processes, the hyperparameters of the optimizer remain unchanged.

[0038] Conditional GAN ​​is introduced to further improve the details of the generated faces. Following the approach of MoFaNeRF, pix2pixHD (Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. High-resolution image synthesis and semantic manipulation with conditional gans. InCVPR, pages 8798–8807, 2018.4) is used as the backbone network of RefineNet. The input of RefineNet is the generated image rendered from the synthesized three-way feature plane. In the training stage, the entire network except RefineNet is first trained, and then RefineNet is trained with the other parts of the network separated. The loss function for training RefineNet is:

[0039]

[0040] in They are generative adversarial network loss, structural similarity loss, and perceptual image quality assessment loss. 1 and μ 2 They are the loss weights corresponding to each loss function, following the settings of pix2pixHD.

[0041] 5. After step 4, a parametric 3D head model that can support 360-degree free-viewing angle rendering can be obtained. Shoot a group of videos, extract the expression coefficient parameter stream of the face in the video based on the expression base, obtain the mesh model of the digital human and the corresponding expression in the video according to the 3DMM model and the expression coefficient, select the texture encoding, and send it to the parametric 3D head model to generate a digital human with dynamic expressions and can be rendered in 360 degrees. The result is as follows: Figure 4 shown.

[0042] This embodiment also uses the model proposed by the present invention to complete the task of fitting a single image. The main idea is to keep the training parameters of the model unchanged and update the potential input space according to the gradient, similar to GAN inversion, but in order to improve the fitting performance, several modifications are made to the model. The main difficulty is that the obtained parameterized three-dimensional head model is trained on a relatively small data set of only 100 characters, so the parameter space of the texture code t is very limited and it is difficult to cover a variety of facial appearances. To address this problem, the neural texture is optimized instead of the latent code or the mapping code w, thereby improving the generalization ability of the fitting. The patch mesh is also optimized by a linear mapping from s to a 3D mesh and a method of connecting them to the rasterization module.

[0043] In the single image fitting task, the L1 loss between the generated frontal view and the input image is used as a constraint. First, the shape encoding s is optimized to obtain a fitted patch mesh that matches the input image. Then the fitted shape is fixed and a neural texture is randomly selected as the initialization to optimize the neural texture. The image is first normalized to align the face of the input image and the facial region of the 3DMM model. The image generated from the patch mesh initialized with the random texture and fitted by the 3D mesh model is used to uniform the skin color with the aligned real image in the wild using Poisson fusion as the preprocessed real image. By optimizing the three-way feature plane space to reduce the difference between the preprocessed image and the image generated by the random texture, the neural texture and the latent code of the hair can be optimized to generate a 360-degree renderable and animatable head to fit the appearance of the input image.

[0044] The 3DMM model obtained by the present invention is defined based on the standard 52 expression bases, which are also used by ARKit. Therefore, the generated or fitted head can be generated by obtaining parameter stream data through a series of expression base media tools. At the same time, some virtual existence phenomena such as hair, mouth and eyelashes that are not tightly attached to the mesh surface and have permanent and consistent appearance features can be generated. Therefore, the present invention proves that the proposed learnable representation method can achieve the performance of complex head models designed by traditional artists and realize high-fidelity animation effects.

[0045] The text-based image conditional diffusion model (Tim Brooks, Aleksander Holynski, and Alexei A. Efros. Instructpix2pix: Learning to follow image editing instructions. In CVPR, pages 18392–18402, 2023.5) is combined with the present invention for 3D head editing. Specifically, after generating or fitting the head, the multi-view image is rendered and edited by a text-guided image-to-image converter. Then, the edited image is used to fine-tune the parameterized three-dimensional head model. The optimizer and learning rate are kept unchanged compared to before during the fine-tuning process. Experience has shown that when fine-tuning the parameter model, sampling in a patch manner on the generated image will achieve better results than the original random sampling strategy. The text-based editing strategy in the present invention aims to add abstract features such as "makeup" and "aging" to the generated head without changing the identity information or appearance information of the character. The edited head can still be animated and driven by a standard expression-based parameter flow.

[0046] Through the above steps, the present invention can directly generate a high-fidelity driven three-dimensional face model from a frontal photo or a video description, and change its hairstyle to generate a result such as Figure 5 Changing the texture or hairstyle of a digital human still has the same video-based expression-driven effect.

[0047] The high-fidelity 360-degree renderable parametric 3D head model generation method based on synthetic data proposed in the present invention realizes the direct generation of a 3D full head model from parameters, reduces the cost of acquiring a parametric 3D model, and contributes to the related research of 3D face generation. The generated 3D face can be applied to multiple fields such as film and television, games, and security. The present invention is the first parametric 3D full head model method that simultaneously realizes 360-degree free view synthesis, image-based inversion, and animation. It not only improves the authenticity and expressiveness of head image generation, but also enhances the model's ability to generate dynamic expressions and multi-view images. Experiments show that facial motion and appearance are well separated in parameter space, achieving state-of-the-art performance in rendering and animation quality.

Claims

1. A method for generating a 360-degree human face head parametric model based on synthetic data, characterized in that: The steps include: S1, obtain several digital humans with different skin colors, hairstyles and appearances, and align their facial regions according to the Prototype Transform; use imaging equipment to shoot the aligned digital humans, select an expression base as the basic expression, and create a high-fidelity synthetic 3D face dataset with expression labels; S2, based on the synthetic 3D face dataset, obtain topologically consistent mesh models of all digital humans under various expressions, that is, models with the same number of vertices and facets; apply the scale transformation, rotation matrix, and translation matrix corresponding to the Prototype transformation to the mesh models, so that they are topologically consistent while ensuring alignment in three-dimensional space; use the principal component analysis method to reduce the dimension of the aligned mesh models with expressions to generate a 3DMM model, and obtain the corresponding 3DMM coefficients; S3, using the synthesized 3D face dataset as a one-hot encoding of the expression dimension, using the 3DMM coefficient as the encoding of the shape dimension, combining the spatially aligned mesh model with expressions, constructing training data, generating a three-dimensional feature plane and a learnable neural texture encoding; constructing and training a parameterized three-dimensional head model, which is used to generate multi-view and multi-expression head images when the 3DMM coefficients are known; S4, constructing data of several digital humans without hair, continuing to fine-tune the parameterized three-dimensional head model obtained in the training step S3, separating the hair from the face; using the text-based image conditional diffusion model, editing the head in the parameterized three-dimensional head model; S5, shoot a group of videos, extract the expression coefficient parameter flow of the face in the video based on the expression base, obtain the mesh model of the digital human and the corresponding expression in the video according to the 3DMM model and the expression coefficient, select the texture coding, and send it into the parameterized three-dimensional human head model trained in step S4 to generate a digital human with dynamic expressions and can be rendered in 360 degrees.

2. The method for generating a 360-degree human face head parametric model based on synthetic data according to claim 1, characterized in that: In step S1, when the aligned digital human is photographed using an imaging device, the position of the digital human in the photographing space is fixed, the distance between the imaging device and the digital human is fixed, and multi-angle shooting is performed at intervals of 15 degrees in the horizontal direction and at 30 degrees downward, 30 degrees upward, and parallel viewing angles in the vertical direction.

3. The method for generating a 360-degree human face and head parametric model based on synthetic data according to claim 1, characterized in that: In step S2, each mesh model has 2548 vertices and 5048 facets.

4. The method for generating a 360-degree human face head parametric model based on synthetic data according to claim 1, characterized in that: In step S3, when generating the three-dimensional feature plane, the grid model is rasterized and projected in six directions, namely, front, back, left, right, up and down, and the planes in the front, back, left, right, up and down directions are connected as one plane to form a new three-plane representation.

5. The method for generating a 360-degree human face head parametric model based on synthetic data according to claim 1, characterized in that: In step S3, the parameterized three-dimensional human head model is trained using the L1 loss function and the weighted generative adversarial network loss: in is the photometric loss function, is the dual GAN ​​loss; is the density regularization term, and λ1 and λ2 are the weights corresponding to each loss function.

6. The method for generating a 360-degree human face head parametric model based on synthetic data according to claim 5, characterized in that: The generative adversarial network includes a generator and a discriminator, a learning rate l and a regularization interval R of the generator and the discriminator. i Related: Among them, l base is the base learning rate.

7. The method for generating a 360-degree human face head parametric model based on synthetic data according to claim 6, characterized in that: In step S4, when fine-tuning the parameterized three-dimensional human head model, the network's loss function, optimizer and learning rate are consistent with those in step S3.

8. The method for generating a 360-degree human face and head parametric model based on synthetic data according to claim 1, characterized in that: In step S4, when fine-tuning the parameterized three-dimensional human head model, a refinement network is established, and the loss function of training the refinement network is: in and They are the generative adversarial network loss, structural similarity loss and perceptual image quality assessment loss; μ1 and μ2 are the loss weights corresponding to each loss function.

Images