A Semantic 3D Face Reconstruction Method Based on RGB-D Images

Abstract

This invention discloses a semantic 3D face reconstruction method based on RGB-D images. The method first uses a depth camera or a mobile phone equipped with a depth camera to capture multiple frames of RGB-D images of the head during rotation as input data. A TSDF model of the face is robustly constructed based on the color and depth data contained in the RGB-D images. Finally, a semantic 3D face mesh model is obtained through 3DMM-based face fitting. This method robustly recovers a semantically defined 3D face model with a fixed number of vertices and consistent topology from the input RGB-D video sequence. The recovered face model is of high quality and can be easily used for subsequent tasks requiring facial semantics, such as facial animation.

Description

Technical Field

[0001] The present invention relates to the field of computer graphics, and in particular to a semantic three-dimensional face reconstruction method based on RGB-D images. Background Art

[0002] 3D facial reconstruction has long been a key research area in computer graphics and computer vision. As a crucial and recognizable part of a person, the face largely reflects their individual characteristics, making 3D facial reconstruction crucial. Reconstructed facial models have a wide range of applications, including in entertainment, education, healthcare, and other industries.

[0003] In the current work of 3D facial reconstruction, there are two common methods, each with its own advantages and disadvantages. First, the face reconstruction method based on 3D deformable models uses facial statistical priors to transform the problem into a parameter estimation problem of the 3D deformable model. The restored face model has semantic information, but the final result is still quite different from the real face geometry. Another algorithm based on deep fusion, represented by the KinectFusion algorithm, can better restore the target geometry, but this is a 3D reconstruction method for general objects or scenes. Directly using it for 3D facial reconstruction still faces some problems. The reconstructed face model does not have semantic information and is prone to contain non-face parts, which is not convenient for subsequent face-related processing. Summary of the Invention

[0004] The purpose of the present invention is to address the deficiencies of the existing technology and provide a semantic three-dimensional face reconstruction method based on RGB-D images.

[0005] The objective of the present invention is achieved through the following technical solution: The present invention provides a semantic three-dimensional face reconstruction method based on RGB-D images, comprising the following steps:

[0006] Step 1: Use a depth camera or a mobile phone with a depth camera to capture multiple frames of RGB-D images when the head is turned as input data;

[0007] Step 2: Based on the color data and depth data contained in the RGB-D image, a TSDF model of the face is robustly constructed;

[0008] Step 3: Based on the face TSDF model constructed in step 2, a semantic three-dimensional face mesh model is obtained through 3DMM-based face fitting.

[0009] Furthermore, the step (2) includes the following sub-steps:

[0010] (2.1) Preprocessing: First, align the RGB image to the depth image, then perform bilateral filtering on the depth image to reduce noise, and then calculate the vertex map and normal map based on the camera intrinsic parameters;

[0011] The bilateral filtering of the depth map is specifically as follows: Assume that the depth map of the input k-th frame is R k , then after bilateral filtering, the image domain The noise reduction depth D at pixel u k (u) is:

[0012]

[0013] in, W p is the normalization constant, is the standard deviation parameter in the Gaussian distribution;

[0014] The vertex graph and normal graph used in the pose estimation based on the camera intrinsic parameters are as follows: Assuming that the camera intrinsic parameter is K, the homogenized coordinates of pixel u are Vertex graph V k Obtained by back-projection in the camera coordinate system:

[0015]

[0016] The normal graph can be obtained by taking the cross product of adjacent vertices in the vertex graph:

[0017] N k (u)=v[(V k (u+1,v)-V0(u,v))×(V k (u,v+1)-V k (u,v))]

[0018] Among them, v[x]=x / |x|2 means normalizing the vector, and the final result is the unit normal vector;

[0019] (2.2) Facial feature point detection and face segmentation: Use the DDE algorithm to detect facial feature points and obtain the two-dimensional coordinates of facial feature points in the RGB image. The set is recorded as Assumptions The corresponding depth in the depth map is d l , get the three-dimensional coordinates of the feature points The relationship is as follows:

[0020]

[0021] Then, the input RGB image is semantically segmented through a neural network, removing parts irrelevant to the face and retaining only the facial data to achieve face segmentation;

[0022] (2.3) Pose estimation: Estimate the six-degree-of-freedom rigid body transformation T of the current RGB-D frame in the global coordinate system by combining the ICP variant method with feature points:

[0023] E icp =w dense E dense +w landmarks E landmarks

[0024] Among them, E dense Represents the error based on the depth image, w dense Represents the weight of the item; E landmarks is the error based on facial feature points, w landmarks Represents the weight of the item;

[0025] E dense The specific formula is:

[0026]

[0027] Among them, v k-1 、n k-1 They are the vertex map and normal map rendered from the previous frame perspective,

[0028] E landmarks The specific formula is:

[0029]

[0030] Among them, T0 represents the rigid body transformation of the first frame, Represents the three-dimensional coordinates of each facial feature point in the first frame, Represents the three-dimensional coordinates of each facial feature point in the kth frame, which is the current frame;

[0031] (2.4) Data fusion: perform weighted averaging of the fused TSDF value and the TSDF value of the current frame;

[0032] (2.5) Surface prediction: Based on the currently fused TSDF model, the predicted vertex map under the current frame perspective is rendered using the ray casting method for frame-to-model pose estimation in the next frame.

[0033] Furthermore, the step (3) includes the following sub-steps:

[0034] (3.1) Template pre-fitting: Based on the 3D feature points of the face detected in the first frame RGB-D image Use the corresponding semantic points in the 3DMM to build one-to-one constraints, and then use the ridge regression method to calculate the 3DMM coefficients to obtain a pre-fitted 3DMM model as the initial template for subsequent non-rigid deformation;

[0035] (3.2) Constructing a deformation map: Embedded deformation is selected as the non-rigid deformation representation. The deformation of the entire face space is defined as a lower-dimensional deformation space determined by the deformation map. The deformation of the entire face template is indirectly controlled by controlling the affine transformation of each deformation vertex. The deformation map is constructed by uniformly sampling on the deformation template. The uniform sampling is performed using the farthest point sampling algorithm.

[0036] Each deformation graph node represents an affine transformation A, which consists of a 3x3 matrix R and a three-dimensional translation vector t, with a total of 12 variables. The spatial position of the node is denoted as g; for a point v in space i , its position after deformation Determined by the affine transformation of its nearest N nodes:

[0037]

[0038] where g j represents the position of node j, w j Indicates the influence weight of node j on the vertex; unnormalized The calculation formula is:

[0039]

[0040] Among them, d max is the distance from node j to the N+1th nearest node; the normalized weight w j for:

[0041]

[0042] (3.3) Non-rigid alignment based on TSDF: The affine transformation A represented by each node of the deformation graph is calculated by minimizing the energy function:

[0043] E fit =E data +w rot E rot +w reg E reg

[0044] Among them, E data Represents the data constraint item, and the calculation formula is:

[0045] E data =w sdf E sdf +w landmarks E landmarks

[0046] E data The distance function term E in the term sdfThe definition is as follows:

[0047]

[0048] Where D(v) represents the TSDF value at vertex v in the pre-fitted 3DMM template, t shell is the cutoff distance value of TSDF, w sdf Represents the distance function term E sdf The weight, w landmarks Represents the facial feature point item E landmarks The weight of

[0049] E data The facial feature point item E in the item landmarks The definition is as follows:

[0050]

[0051] in, Represents the three-dimensional feature points of the face, Q l Indicates the corresponding feature points in the pre-fitted 3DMM template;

[0052] The rotation constraint consists of the rotation energy terms for all nodes in the deformation graph:

[0053]

[0054] For each node in the affine transformation R, c1, c2, c3 are the three-dimensional column vectors of the third-order R matrix, and the rotation energy term of each node is defined as:

[0055] Rot(R)=(c1·c2) 2 +(c1·c3) 2 +(c2·c3) 2 +(c1·c1-1) 2 +(c2·c2-1) 2 +(c3·c3-1) 2

[0056] Regularization constraint E reg It is used to ensure that the deformations represented by adjacent nodes are consistent, and plays a role in smoothing the deformations represented by adjacent nodes. It is defined as:

[0057]

[0058] in, represents the set of adjacent nodes of node j in the deformation graph, g n Indicates the location of n nodes, t n Represents the translation vector in the affine transformation represented by the n-node.

[0059] The present invention generally designs an iterative optimization strategy from coarse to fine. In the initial iteration round, w is assigned reg A larger value, and gradually decreases with the increase of iteration rounds w reg Compared with directly minimizing the energy function, this iterative optimization strategy from coarse to fine can avoid the problem from falling into the local optimal solution, which makes it difficult to obtain the optimal fitting result. In each iteration, E is minimized. fit The energy function is a highly nonlinear least squares problem, solved using the Gauss-Newton method. The final optimization solution is the affine transformation represented by each node in the deformation graph. Based on the non-rigid transformation represented by the deformation graph, the pre-fitted 3DMM template is non-rigidly deformed to obtain the final semantic 3D face model.

[0060] The beneficial effect of the present invention is that it makes full use of the face prior, combines the face feature points and the face segmentation method, makes improvements on the basis of the original fusion algorithm, and designs a fusion method specifically for face reconstruction, which can reconstruct the TSDF model of the face more efficiently and robustly. In addition, the present invention proposes a method for fitting the 3DMM template directly based on the TSDF representation of the face, which can avoid explicitly looking for the nearest point as the corresponding point, and uses the deformation map as a non-rigid deformation representation, while reducing the complexity of the problem. It can robustly handle the situation where the input data is incomplete. Finally, the algorithm proposed by the present invention can robustly reconstruct a semantically three-dimensional face model with a fixed number of vertices and consistent topology from the input RGB-D video. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 This is a flowchart of the semantic 3D face reconstruction method based on RGB-D images. DETAILED DESCRIPTION

[0062] The present invention will be described in detail below with reference to the accompanying drawings.

[0063] The present invention provides a semantic 3D face reconstruction method based on RGB-D images, comprising the following steps:

[0064] Step 1: The user uses a depth camera or a mobile phone with a depth camera to capture multiple frames of RGB-D images while turning the head as input data.

[0065] Step 2: Based on the color data and depth data contained in the input RGB-D image, a TSDF (Truncated Signed Distance Function) model of the face is robustly constructed.

[0066] This step is the core of the present invention and is divided into the following sub-steps, which process each frame of the input RGB-D image as follows.

[0067] (1) Preprocessing

[0068] The first step in the preprocessing phase is to align the depth image with the RGB image, and align the RGB image to the depth image. The present invention first performs bilateral filtering on the depth image to reduce noise. The specific steps of bilateral filtering are as follows: Assume that the depth image of the input k-th frame is R k , then after bilateral filtering, the image domain The noise reduction depth D at pixel u k (u) is:

[0069]

[0070] in, W p is the normalization constant, is the standard deviation parameter in the Gaussian distribution.

[0071] Then, the vertex graph and normal graph used in pose estimation are calculated based on the camera intrinsic parameters. Assuming that the camera intrinsic parameter is K, the homogenized coordinates of pixel u are Vertex graph V k Obtained by back-projection in the camera coordinate system:

[0072]

[0073] Correspondingly, the normal graph can be obtained by the cross product of adjacent vertices in the vertex graph:

[0074] N k (u)=v[(V k (u+1,v)-V k (u,v))×(V k (u,v+1)-V k (u,v))]

[0075] Among them, v[x]=x / |x|2 means normalizing the vector, and the final result is the unit normal vector.

[0076] (2) Facial feature point detection and face segmentation.

[0077] The present invention performs facial feature point detection and face segmentation on the input RGB image. The facial feature point detection method used in the present invention is the DDE algorithm (Displaced Dynamic Expression). The specific facial feature point detection method is referenced in "Cao C, Hou Q, Zhou K. Displaced dynamic expression regression for real-time facial tracking and animation [J]. ACM Transactions on graphics (TOG), 2014, 33 (4): 1-10.", thereby obtaining the two-dimensional coordinates of 75 facial feature points in the RGB image, and the set thereof is recorded as As in the back projection in sub-step 1, assume The corresponding depth in the depth map is d l , we can get the three-dimensional coordinates of the feature points The relationship is as follows:

[0078]

[0079] The present invention also performs face segmentation on the input RGB image. Face segmentation is achieved through a neural network. The network structure adopted in the present invention is BiSeNet《Yu C, Wang J, Peng C, et al. BiSeNet: Bilateral segmentation network for real-time semantic segmentation[C]. in: Proceedings of the European conference on computer vision (ECCV). 2018: 325-341.》, which is trained on the CelebAMask-HQ《Lee CH, Liu Z, Wu L, et al. MaskGAN: Towards Diverse and Interactive FacialImage Manipulation[C]. in: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2020.》dataset. Then, the trained network can be used to perform semantic segmentation on the input RGB image, eliminating parts unrelated to the face, such as background, clothes, etc., and retaining only the facial data. For the specific network training and inference process, please refer to "Yu C, Wang J, Peng C, et al. BiSeNet: Bilateral segmentation network for real-time semantic segmentation [C]. in: Proceedings of the European conference on computer vision (ECCV). 2018: 325-341."

[0080] (3) Pose estimation.

[0081] Estimate the six-degree-of-freedom rigid body transformation T of the current RGB-D frame in the global coordinate system for use in the next step of data fusion. The present invention proposes an ICP (Iterative Closest Point) variant method that combines feature points.

[0082] E icp =w dense E dense +w landmarks E landmarks

[0083] Among them, E denseRepresents the error based on the depth image, E landmarks is the error based on facial feature points. E dense The error term is similar to the error term used by the ICP algorithm in KinectFusion《Newcombe RA, Izadi S, Hilliges O, et al. KinectFusion: Real-time dense surface mapping and tracking[C].in: Proceedings of 2011 10th IEEE international symposium on mixed and augmented reality.2011:127-136.》. It uses a frame-to-model tracking method. Instead of directly aligning the current frame data with the original data of the previous frame, it aligns it with the image rendered by the current fused model in the previous frame’s perspective. The vertex map and normal map rendered in the previous frame’s perspective are denoted as v k-1 、n k-1 For the error term between each pair of corresponding points, the present invention selects the distance from the point to the plane as the error metric. dense The specific formula of the term is:

[0084]

[0085] For the specific task of face reconstruction, the present invention also uses an energy function term E based on effective face feature points. landmarks , the specific formula is:

[0086]

[0087] Among them, T0 represents the rigid body transformation of the first frame, Represents the three-dimensional coordinates of each facial feature point in the first frame, Represents the three-dimensional coordinates of each facial feature point in the kth frame, which is the current frame.

[0088] The present invention proposes to minimize E icp This is still a least squares problem, so the normal equations can be constructed on the GPU, and then the linear equations can be solved on the CPU to quickly solve the problem. For specific solution methods, please refer to "Whelan T, Leutenegger S, Salas-Moreno R, et al. ElasticFusion: Dense SLAM without a posegraph [C]. in: Proceedings of Robotics: Science and Systems. 2015."

[0089] (4)Data fusion.

[0090] Once the pose estimation is completed, the depth data aligned to the global coordinate system can be fused into the TSDF volume. The specific fusion method is to update the TSDF voxels within a certain stage distance and perform a weighted average of the fused TSDF value and the TSDF value in the current frame. For specific methods, please refer to "Newcombe RA, Izadi S, Hilliges O, et al. Kinectfusion: Real-time dense surface mapping and tracking[C]. in: Proceedings of 2011 10th IEEE international symposium on mixed and augmented reality. 2011: 127-136."

[0091] (5) Surface prediction.

[0092] Based on the currently fused TSDF model, a ray casting method is used to render the predicted vertex map for the current frame's perspective for frame-to-model pose estimation for the next frame. For specific methods, please refer to "Newcombe RA, Izadi S, Hilliges O, et al. Kinectfusion: Real-time dense surface mapping and tracking [C]. in: Proceedings of 2011 10th IEEE international symposium on mixed and augmented reality. 2011: 127-136."

[0093] Step 3: Based on the face TSDF model obtained in step 2, a semantic 3D face mesh model is obtained by face fitting based on 3DMM (3D Morphable Model).

[0094] (1) Template pre-fitting.

[0095] The present invention detects three-dimensional feature points of the face in the first frame RGB-D image. Use the corresponding semantic points in 3DMM to build one-to-one constraints, and then use the ridge regression method [6]To calculate the 3DMM coefficients, a pre-fitted 3DMM model is obtained as the initial template for subsequent non-rigid deformation. For specific methods, please refer to "Zhu X, Lei Z, Yan J, et al. High-fidelity pose and expression normalization for face recognition in the wild[C]. in: Proceedings of the IEEE conference on computer vision and pattern recognition. 2015: 787-796."

[0096] In addition, because the TSDF reconstructed in step 2 is scaled in actual physical units, while the 3DMM template is not necessarily so, the scale of the 3DMM template needs to be adjusted according to actual conditions to conform to the TSDF model obtained in step 2.

[0097] (2) Construct a deformation map.

[0098] The present invention selects the embedded deformation proposed by Sumner et al. [7] As a non-rigid deformation representation, the deformation of the entire face space is defined as a lower-dimensional deformation space determined by the deformation map. By controlling the affine transformation of each deformation vertex, the deformation of the entire face template is indirectly controlled. This paper chooses to construct the deformation map by uniformly sampling the deformation template, implementing the Farthest Point Sampling (FPS) algorithm for uniform sampling.

[0099] Each deformation graph node represents an affine transformation A, which consists of a 3x3 matrix R and a three-dimensional translation vector t, a total of 12 variables, and the spatial position of the node is recorded as g. i , its position after deformation Determined by the affine transformation of its nearest N nodes:

[0100]

[0101] where g j represents the position of node j, w j Indicates the influence weight of node j on the vertex. The calculation formula is:

[0102]

[0103] Among them, d max is the distance from node j to the N+1th nearest node. The normalized weight wj for:

[0104]

[0105] (3) Non-rigid alignment based on TSDF.

[0106] The present invention designs a novel energy function, by minimizing the energy function E fit , to calculate the affine transformation A represented by each node of the deformation graph:

[0107] E fit =E data +w rot E rot +w reg E reg

[0108] Among them, E data Represents the data constraint item, and the calculation formula is:

[0109] E data =w sdf E sdf +w landmarks E landmarks

[0110] E data The distance function term E in the term sdf The definition is as follows:

[0111]

[0112] Where D(v) represents the TSDF value at vertex v in the pre-fitted 3DMM template, t shell is the cutoff distance value of TSDF;

[0113] E data The facial feature point item E in the item landmarks The definition is as follows:

[0114]

[0115] in, Indicates the three-dimensional feature points of the face detected in the previous chapter, Q l Indicates the corresponding feature points in the pre-fitted 3DMM template.

[0116] The rotation constraint consists of the rotation energy terms for all nodes in the deformation graph:

[0117]

[0118] For each node in the affine transformation R, c1, c2, c3 are the three-dimensional column vectors of the third-order R matrix, and the rotation energy term of each node is defined as:

[0119] Rot(R)=(c1·c2) 2 +(c1·c3) 2 +(c2·c3) 2 +(c1·c1-1) 2 +(c2·c2-1) 2 +(c3·c3-1) 2

[0120] Regularization constraint E reg It is used to ensure that the deformations represented by adjacent nodes are consistent, and plays a role in smoothing the deformations represented by adjacent nodes. It is defined as:

[0121]

[0122] in, Represents the set of adjacent nodes of node j in the deformation graph.

[0123] The present invention generally designs an iterative optimization strategy from coarse to fine. In the initial iteration round, w is assigned reg A larger value, and as the number of iterations increases w reg Each time it is halved. In each iteration, E is minimized fit The energy function is a highly nonlinear least-squares problem, solved using the Gauss-Newton method. The final optimization solution is the affine transformation A represented by each node in the deformation graph. Based on the non-rigid transformation represented by the deformation graph, the pre-fitted 3DMM template is non-rigidly deformed. The non-rigid deformation process is described in sub-step 2, ultimately resulting in a semantically defined 3D face model.

Claims

1. A semantic 3D face reconstruction method based on RGB-D images, characterized in that: The following steps are involved: Step 1: Use a depth camera or a mobile phone with a depth camera to capture multiple frames of RGB-D images when the head is turned as input data; Step 2: Based on the color data and depth data contained in the RGB-D image, a TSDF model of the face is robustly constructed; Step 3: Based on the face TSDF model constructed in step 2, a semantic 3D face mesh model is obtained through face fitting based on 3DMM; The step (2) includes the following sub-steps: (2.1) Preprocessing: First, align the RGB image to the depth image, then perform bilateral filtering on the depth image to reduce noise, and then calculate the vertex map and normal map based on the camera intrinsic parameters; (2.2) Facial feature point detection and face segmentation: First, the DDE algorithm is used to detect facial feature points and obtain the two-dimensional coordinates of facial feature points in the RGB image. The set is recorded as Assumptions The corresponding depth in the depth map is d l , get the three-dimensional coordinates of the feature points The relationship is as follows: Then, the input RGB image is semantically segmented through a neural network, removing parts irrelevant to the face and retaining only the facial data to achieve face segmentation; (2.3) Pose estimation: Estimate the six-degree-of-freedom rigid body transformation T of the current RGB-D frame in the global coordinate system by combining the ICP variant method with feature points: HAVE BEEN icp =w dense HAVE BEEN dense +w landmarks HAVE BEEN landmarks Among them, E dense Represents the error based on the depth image, w dense Represents the weight of the item; E landmarks is the error based on facial feature points, w landmarks Represents the weight of the item; E dense The specific formula is: Among them, v k-1 、n k-1 They are the vertex map and normal map rendered from the previous frame perspective, E landmarks The specific formula is: Among them, T0 represents the rigid body transformation of the first frame, Represents the three-dimensional coordinates of each facial feature point in the first frame, Represents the three-dimensional coordinates of each facial feature point in the kth frame, which is the current frame; (2.4) Data fusion: perform weighted averaging of the fused TSDF value and the TSDF value of the current frame; (2.5) Surface prediction: Based on the currently fused TSDF model, the predicted vertex map under the current frame perspective is rendered using the ray casting method for frame-to-model pose estimation in the next frame.

2. The semantic 3D face reconstruction method based on RGB-D images according to claim 1, characterized in that: In the step (2.1), the bilateral filtering of the depth map is specifically as follows: assuming that the depth map of the input k-th frame is R k , then after bilateral filtering, the image domain The noise reduction depth D at pixel u k (u) is: in, W p is the normalization constant, is the standard deviation parameter in the Gaussian distribution.

3. The semantic 3D face reconstruction method based on RGB-D images according to claim 1, characterized in that: In the step (2.1), the vertex graph and normal graph are calculated according to the camera intrinsic parameters as follows: Assuming that the camera intrinsic parameter is K, the homogenized coordinates of pixel u are Vertex graph V k Obtained by back-projection in the camera coordinate system: The normal graph can be obtained by taking the cross product of adjacent vertices in the vertex graph: N k (u)=v[(V k (u+1,v)-V k (u,v))×(V k (u,v+1)-V k (u,v))] Among them, v[x]=x / |x|2 means normalizing the vector, and the final result is the unit normal vector.

4. The semantic 3D face reconstruction method based on RGB-D images according to claim 1, characterized in that: The step (3) includes the following sub-steps: (3.1) Template pre-fitting: Based on the 3D feature points of the face detected in the first frame RGB-D image Use the corresponding semantic points in the 3DMM to build one-to-one constraints, and then use the ridge regression method to calculate the 3DMM coefficients to obtain a pre-fitted 3DMM model as the initial template for subsequent non-rigid deformation; (3.2) Constructing a deformation graph: Embedded deformation is selected as the non-rigid deformation representation. The deformation of the entire face space is defined as a lower-dimensional deformation space determined by the deformation graph. The deformation of the entire face template is indirectly controlled by controlling the affine transformation of each deformation vertex. Perform uniform sampling on the deformation template to construct the deformation map, and perform uniform sampling using the farthest point sampling algorithm; Each deformation graph node represents an affine transformation A, which consists of a 3x3 matrix R and a three-dimensional translation vector t, with a total of 12 variables. The spatial position of the node is denoted as g; for a point v in space i , its position after deformation Determined by the affine transformation of its nearest N nodes: where g j represents the position of node j, w j Indicates the influence weight of node j on the vertex; unnormalized The calculation formula is: Among them, d max is the distance from node j to the N+1th nearest node; the normalized weight w j for: (3.3) Non-rigid alignment based on TSDF: Calculate the affine transformation A represented by each node of the deformation graph by minimizing the energy function: HAVE BEEN fit =E data +w rot HAVE BEEN rot +w reg HAVE BEEN reg Among them, E data Represents the data constraint item, and the calculation formula is: HAVE BEEN data =w sdf HAVE BEEN sdf +w landmarks HAVE BEEN landmarks E data The distance function term E in the term sdf The definition is as follows: Where D(v) represents the TSDF value at vertex v in the pre-fitted 3DMM template, t shell Indicates the cutoff distance value of TSDF, w sdf Represents the distance function term E sdf The weight, w landmarks Represents the facial feature point item E landmarks The weight of E data The facial feature point item E in the item landmarks The definition is as follows: in, Represents the three-dimensional feature points of the face, Q l Indicates the corresponding feature points in the pre-fitted 3DMM template; The rotation constraint consists of the rotation energy terms for all nodes in the deformation graph: For each node in the affine transformation R, c1, c2, c3 are the three-dimensional column vectors of the third-order R matrix, and the rotation energy term of each node is defined as: Rot(R)=(c1·c2) 2 +(c1·c3) 2 +(c2·c3) 2 +(c1·c1-1) 2 +(c2·c2-1) 2 +(c3·c3-1) 2 Regularization constraint E reg It is used to ensure that the deformations represented by adjacent nodes are consistent, and plays a role in smoothing the deformations represented by adjacent nodes. It is defined as: in, represents the set of adjacent nodes of node j in the deformation graph, g n Indicates the location of n nodes, t n Represents the translation vector in the affine transformation represented by the n-node.

Images