Technology for acquiring 3D (three-dimensional) information of human face and head from video

Abstract

The invention relates to the technical field of information processing, and discloses a technology for acquiring 3D information of a human face and a head from a video. Comprising the steps of learning head 3D information from different appearance videos, estimating a head geometric structure by using a FLAME model, representing modeled head geometry by sharing a deformation field and standardizing spatial nerve voxels, introducing learnable geometry attachment features and an appearance condition density formula, and the like. According to the technology, different appearances of the same object can be learned at the same time, training is carried out in a unified network, a vivid head Avatar image is generated, and controllability is achieved. The method has the advantages that multi-appearance learning, vivid rendering and high controllability are achieved, the method is superior to a reference method in the aspect of image synthesis quality, the model efficiency is high, the method can be well generalized to novel head postures and facial expressions, the accuracy of appearance details is kept, and the method is suitable for meeting the requirements for accurate head three-dimensional data and real-time reconstruction in the fields of medical diagnosis, human-computer interaction and the like.

Description

Technical Field

[0001] The present invention relates to the field of information processing technology, and in particular to a technology for obtaining 3D information of a human face and head from a video. Background Art

[0002] With the development of computer vision and artificial intelligence, the recognition and analysis of human faces have become increasingly important. For medical diagnosis, human-computer interaction and other fields, accurate head 3D data and real-time reconstruction are indispensable. However, most existing methods require the collection and recording of individuals in a controlled environment, which is difficult to adapt to the fast real-time requirements in large-scale scenarios. To solve this problem, we can use a variety of technologies to assist in the automatic pixel-level reconstruction of human faces to meet various practical needs. For example, the 3D position coordinates of the target object can be directly detected using devices such as depth cameras or laser scanners, or the required data can be obtained indirectly through manual marking; for the recognition and positioning of multiple target objects in complex environments, we can use machine learning algorithms combined with image data analysis and understanding for automatic classification, thereby improving work efficiency while ensuring accuracy. Finally, cost factors and related safety measures should also be considered to ensure the long-term stable and reliable operation of the entire system. In summary, any task involving face recognition, analysis and reconstruction, whether in medical or commercial fields, requires comprehensive consideration of all the above factors to achieve better results. Summary of the invention

[0003] In view of the shortcomings of the prior art, the present invention solves the technical problems by adopting a technical solution: a technology for obtaining 3D information of a face and head from a video, the steps are as follows:

[0004] Input Video: Learn head 3D information from video clips with different appearances.

[0005] Head geometry estimation: Use the FLAME model to estimate the 3D head geometry, including head pose and facial expression.

[0006] Personalized Deformable NeRF: Using a shared canonical space and deformable NeRF, we model the head geometry and generate images through volume rendering.

[0007] Learned Appearance Embeddings: Define a set of learned appearance embeddings and attach them to the vertices of a 3DMM mesh for appearance-conditioned rendering.

[0008] Appearance Conditional NeRF: We incorporate the learned appearance embeddings into NeRF in the canonical space for predicting color and density and generating realistic rendered images.

[0009] The technical concept of this method can be summarized as follows:

[0010] Shared deformation fields and canonical space neural voxel representations:

[0011] To solve the problem that existing head NeRF methods can only model videos with a specific appearance, this method introduces a shared deformation field and a canonical space neural voxel representation. By learning a shared deformation field, this method can deform video frames with different appearances into the same canonical space, thereby eliminating the influence of appearance differences. In the canonical space, this method uses a neural voxel representation to model head geometry and appearance, so that it can learn multi-appearance head NeRF.

[0012] Use deformation gradient ^F x To normalize, the formula is as follows:

[0013] X c =^F x

[0014] Among them, X c is the sampling point in the canonical space, x is the sampling point in the observation space, ^F x is the deformation gradient. This formula represents the sampling point x in the observation space through the deformation gradient ^F x Transform into the canonical space to eliminate the influence of appearance differences.

[0015] Learnable geometric attachment features:

[0016] In order to distinguish features of different appearances, this method introduces learnable geometric attachment features. These features are attached to the vertices of the 3DMM mesh and deform with the deformation of the mesh. In this way, this method can remember local changes and better handle appearance changes. The invention defines a set of learnable latent features in UV space: Z = Z i N I = 1, corresponding to the N appearances in the video collection. These features are attached to the vertices of the 3DMM mesh and sampled by the nearest triangle search. The sampled features Z x is used to condition the NeRF prediction, thus affecting the color and density estimation. Sampled feature Z x The formula is:

[0017]

[0018] Where Zx is the sampled feature, wi is the weight of the i-th vertex, and z(vi) is the feature of the i-th vertex.

[0019] Appearance condition density formula:

[0020] In order to predict color and density more accurately, this method designs an appearance-conditional density formula. This formula combines shared density and appearance-dependent density shift, so that it can simultaneously predict appearance-dependent density change and shared density. The appearance-dependent density shift is predicted by appearance embedding and appearance-conditional density shift function, while the shared density is predicted by the neural voxel representation in the canonical space.

[0021] Iθ,e,Z j =F(θ,e,Z j , K) represents the formula for synthesizing the target image of the present invention, where θ is the head posture, e is the facial expression, and Z j is the appearance embedding, and K is the camera intrinsic parameter.

[0022] σ(r(t))·c(r(t))·T(t)dt represents the formula for rendering different iable voxels, where r(t) is the camera ray, σ(t) is the density, c(t) is the radiance, and T(t) is the accumulated transmittance.

[0023] Single unified network:

[0024] Existing head NeRF methods require training models for each appearance separately, resulting in low training efficiency. This method uses a single unified network to learn dynamically deformable head NeRFs for multiple appearances, without the need to train models for each appearance separately, thus improving training efficiency.

[0025] Appearance Condition Technology:

[0026] In order to better handle appearance changes, this method uses appearance embedding and appearance conditional density formula to incorporate appearance-related information into the neural voxel representation. This helps this method better handle appearance changes and improves rendering quality. The neural voxel representation is used in the canonical space to model the head geometry and appearance.

[0027]

[0028]

[0029]

[0030] in is the characteristic of the sampling point in the canonical space, σ(xc) is the predicted density value, c(xc, d i , σ(xc)) is the predicted color value. This formula means that the features of the sampling point in the canonical space are input into the MLP to predict the density and color of the point respectively.

[0031] The beneficial effects of the present invention are as follows:

[0032] Multi-appearance learning: This technique is able to simultaneously learn different appearances of the same object and train them in a unified network, avoiding the tedious process of training a separate model for each appearance.

[0033] Photorealistic rendering: This technology is able to generate head avatar images with high fidelity and realism, including details such as facial expressions and hair.

[0034] Controllability: This technology can generate head avatar images with specific appearance by controlling head pose, facial expression, and appearance embedding BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 It is a flow chart of the present invention DETAILED DESCRIPTION

[0036] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. The embodiments of the present invention are provided for the purpose of illustration and description, and are not intended to be exhaustive or to limit the present invention to the disclosed forms. Many modifications and variations will be apparent to those of ordinary skill in the art. The embodiments are selected and described in order to better illustrate the principles and practical applications of the present invention, and to enable those of ordinary skill in the art to understand the present invention and thereby design various embodiments with various modifications suitable for specific uses.

[0037] Example:

[0038] See also Figure 1 The present invention provides a technical solution: a technology for obtaining 3D information of a human face and head from a video, comprising the following steps:

[0039] Step 1: Input video: learn head 3D information from video clips with different appearances;

[0040] Step 2: Head set estimation: Use the FLAME model to estimate the 3D geometry of the head, including head pose and facial expression;

[0041] Step 3: Personalized deformable NeRF: Use the shared canonical space and deformable NeRF to achieve head geometry modeling and generate images through volume rendering;

[0042] Step 4, learned appearance embeddings: define a set of learned appearance embeddings and attach them to the vertices of the 3DMM mesh for appearance-conditioned rendering;

[0043] Step 5, appearance-conditioned NeRF: The learned appearance embeddings are integrated into NeRF in the canonical space to predict color and density and generate realistic rendered images.

[0044] Head geometry modeling is achieved by introducing a shared deformation field and canonical space neural voxel representation. By learning a shared deformation field, video frames with different appearances are deformed into the same canonical space to eliminate the impact of appearance differences.

[0045] In canonical space, a neural voxel representation is used to model head geometry and appearance, using deformation gradients^F x To normalize, the formula is as follows:

[0046] X c =^F x ;

[0047] Where X c is the sampling point in the canonical space, x is the sampling point in the observation space, ^F x is the deformation gradient.

[0048] We define a set of learned appearance embeddings that are attached to the vertices of a 3DMM mesh via learnable geometric attachment features that deform as the mesh deforms.

[0049] In UV space, we define a set of learnable latent features, Z = Z i N I = 1, corresponding to the N appearances in the video set, the features are attached to the vertices of the 3DMM mesh and sampled by the nearest triangle search to obtain the sampled features Z x .

[0050] The sampled feature Z x The formula is:

[0051]

[0052] Among them, Z x is the sampled feature, w i is the weight of the ith vertex, z(v i ) is the feature of the i-th vertex.

[0053] Using appearance embedding and appearance conditional density formula, appearance-related information is integrated into the neural voxel representation, and shared density and appearance-related density offset are combined to simultaneously predict appearance-related density changes and shared density. Appearance-related density offset is predicted by appearance embedding and appearance conditional density offset function. The prediction formula is as follows:

[0054] Iθ,e,Z j =F(θ, e, Z j , K);

[0055] The formula for synthesizing the target image is represented by, where θ is the head posture, e is the facial expression, and Z jis the appearance embedding, K is the camera intrinsic parameter;

[0056] The shared density is predicted by the neuron voxels in the canonical space using the following formula:

[0057] σ(r(t))·c(r(t))·T(t)dt;

[0058] Formulas representing different voxel renderings, where r(t) is the camera ray, σ(t) is the density, c(t) is the radiance, and T(t) is the accumulated transmittance.

[0059] The head geometry and appearance are modeled using a neural voxel representation in the canonical space. The features of the sampled points in the canonical space are input into the MLP to predict the density and color of the point, respectively. The prediction formula is as follows:

[0060]

[0061]

[0062]

[0063] in is the characteristic of the sampling point in the canonical space, σ(xc) is the predicted density value, c(xc,d i , σ(xc)) is the predicted color value.

[0064] The final validation,experimental approach,is evaluated using the VidCol dataset: it contains head video clips of,different subjects with different appearances, which is used to evaluate the,performance of the proposed technique on novel head poses and,facial expression synthesis.

[0065] Confirmatory test:

[0066] Ablation studies: By removing certain components of the technique, such as appearance embedding and density shift, we evaluate how these components affect the quality of image synthesis.

[0067] Qualitative and Quantitative Evaluation: The images generated by the proposed technique are qualitatively and quantitatively evaluated using metrics such as LPIPS, DISTS, SSIM, PSNR, and FID, and compared with baseline methods.

[0068] Test results:

[0069] The technology significantly outperforms the baseline method in terms of image synthesis quality, and is able to generate more realistic and realistic head avatar images. The model efficiency of the technology is higher than the baseline method, and it can be trained in a shorter time. The technology can generalize well to novel head poses and facial expressions, and maintain the accuracy of appearance details.

[0070] Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field and related fields without creative work should fall within the scope of protection of the present invention. The structures, devices and operating methods not specifically described and explained in the present invention are implemented according to the conventional means in the field unless otherwise specified and limited.

Claims

1. A technology for obtaining 3D information of a human face and head from a video, characterized in that: The following steps are involved: Step 1: Input video: learn head 3D information from video clips with different appearances; Step 2: Head set estimation: Use the FLAME model to estimate the 3D geometry of the head, including head pose and facial expression; Step 3: Personalized deformable NeRF: Use the shared canonical space and deformable NeRF to achieve head geometry modeling and generate images through volume rendering; Step 4, learned appearance embeddings: define a set of learned appearance embeddings and attach them to the vertices of the 3DMM mesh for appearance-conditioned rendering; Step 5, appearance-conditioned NeRF: The learned appearance embeddings are integrated into NeRF in the canonical space to predict color and density and generate realistic rendered images.

2. The technology for obtaining 3D information of a human face and head from a video according to claim 1, characterized in that: Head geometry modeling is achieved by introducing a shared deformation field and canonical space neural voxel representation. By learning a shared deformation field, video frames with different appearances are deformed into the same canonical space to eliminate the impact of appearance differences.

3. The technology for obtaining 3D information of a human face and head from a video according to claim 2, characterized in that: In canonical space, a neural voxel representation is used to model head geometry and appearance, using deformation gradients^F x To normalize, the formula is as follows: X c =^F x ; Where X c is the sampling point in the canonical space, x is the sampling point in the observation space, ^ F x is the deformation gradient.

4. The technology for obtaining 3D information of a human face and head from a video according to claim 1, characterized in that: We define a set of learned appearance embeddings that are attached to the vertices of a 3DMM mesh via learnable geometric attachment features that deform as the mesh deforms.

5. The technology for obtaining 3D information of a human face and head from a video according to claim 4, characterized in that: In UV space, we define a set of learnable latent features, Z = Z i N I = 1, corresponding to the N appearances in the video set, the features are attached to the vertices of the 3DMM mesh and sampled by the nearest triangle search to obtain the sampled features Z x .

6. The technology for obtaining 3D information of a human face and head from a video according to claim 5, characterized in that: The sampled feature Z x The formula is: Among them, Z x is the sampled feature, w i is the weight of the ith vertex, z(v i ) is the feature of the i-th vertex.

7. The technology for obtaining 3D information of a human face and head from a video according to claim 3, characterized in that: Using appearance embedding and appearance conditional density formula, appearance-related information is integrated into the neural voxel representation, and shared density and appearance-related density offset are combined to simultaneously predict appearance-related density changes and shared density. Appearance-related density offset is predicted by appearance embedding and appearance conditional density offset function. The prediction formula is as follows: Iθ,e,Z j =F(θ,e,Z j ,K); The formula for synthesizing the target image is represented by, where θ is the head posture, e is the facial expression, and Z j is the appearance embedding, K is the camera intrinsic parameter; The shared density is predicted by the neuron voxels in the canonical space using the following formula: σ(r(t))·c(r(t))·T(t)dt; Formulas representing different voxel renderings, where r(t) is the camera ray, σ(t) is the density, c(t) is the radiance, and T(t) is the accumulated transmittance.

8. The technology for obtaining 3D information of a human face and head from a video according to claim 7, characterized in that: The head geometry and appearance are modeled using a neural voxel representation in the canonical space. The features of the sampled points in the canonical space are input into the MLP to predict the density and color of the point, respectively. The prediction formula is as follows: in is the characteristic of the sampling point in the canonical space, σ(xc) is the predicted density value, c(xc, d i , σ(xc)) is the predicted color value.

Images