Multi-view face reconstruction method based on parametric model

Abstract

The application relates to a multi-view face reconstruction method based on a parameterized model, and comprises the following steps: S1, a plurality of two-dimensional face images of different angles are acquired and processed, and then a face key point detector Dlib algorithm is used for face feature point detection and tracking; S2, according to the feature point positions in the two-dimensional face images of different angles, mask cutting of unobstructed areas of a front face, a left side face and a right side face is carried out respectively; S3, the mask cutting is input into a Resnet network, and different-angle face shapes, expressions, postures, illuminations and camera parameters are regressed; S4, parameters are fitted to a parameterized face model, and corresponding three-dimensional face models are generated; S5, texture sampling is carried out, and a differentiable renderer is used to generate rendered pictures; and S6, loss calculation is carried out on the rendered pictures and original pictures, parameters are back propagated, and a three-dimensional face model is optimized. The application can restore a more realistic face model, and effectively improves the face quality in a complex scene.

Description

Technical Field

[0001] This invention relates to the field of face reconstruction, and more specifically to a multi-view face reconstruction method based on a parametric model. Background Technology

[0002] 3D face reconstruction technology plays a crucial role in fields such as face recognition, facial animation, virtual reality, 3D games, and medicine. Currently, the following problems exist:

[0003] First, there is the diversity of facial expressions. Human faces possess a wealth of expressive information. Psychologists initially believed that humans have six basic expressions: anger, joy, shock, sadness, fear, and disgust. In addition, there are 15 complex expressions. The positions of facial landmarks and 3D reconstruction points change accordingly under different facial expressions.

[0004] Second, occlusion by objects. People may wear jewelry, glasses, or have their own body hair, which can obscure part of their facial features and contours, making it more difficult to reconstruct the shape and features of a three-dimensional face.

[0005] Third, the overall pose of the face. Facial keypoint annotation can take two different forms: the first is visible facial edge keypoints, where the position and semantic information of these points change with pose, and all facial keypoints are visible. The second is pseudo-3D perspective labeling, where self-occlusion of the face can cause some keypoints to be invisible, easily leading to errors in the labeled facial information and making it impossible to extract features from these areas. This makes 3D face reconstruction a challenging task.

[0006] Fourth, interference from the surrounding environment. In natural environments, background information in facial images can affect facial information, and uneven lighting can also cause loss of facial texture information, leading to a decrease in the accuracy of predicted 3D points.

[0007] Fifth, there is currently a shortage of 3D face datasets, mainly due to the use of synthetic data and weakly supervised or unsupervised learning methods. Using synthetic data expands the training set to some extent, but there is often a significant difference between synthetic and real face images, leading to a marked weakening of the model's generalization ability and poor training results on real face images. Currently, weakly supervised or unsupervised methods are more commonly used for training, regressing face parameters from facial landmarks or unlabeled 2D face images. Summary of the Invention

[0008] In view of this, the purpose of this invention is to provide a multi-view face reconstruction method based on a parametric model, which can recover a more realistic face model and effectively improve the face quality in complex scenes.

[0009] To achieve the above objectives, the present invention adopts the following technical solution:

[0010] A multi-view face reconstruction method based on a parametric model includes the following steps:

[0011] Step S1: Acquire and process several two-dimensional face images of the same person from different perspectives, and then use the Dlib algorithm of the face landmark detector to detect and track facial feature points;

[0012] Step S2: Based on the location of feature points in the 2D face images from different perspectives, mask the unobstructed areas of the frontal face, left side face, and right side face respectively;

[0013] Step S3: Input the mask image obtained in step S2 into the ResNet network to regress the face shape, expression, pose, lighting and camera parameters from different perspectives;

[0014] Step S4: Fit the face shape, expression, pose, lighting and camera parameters obtained in step S3 to the parametric face model to generate the corresponding 3D face model;

[0015] Step S5: Perform texture sampling, and then use a differentiable renderer to generate the rendered image;

[0016] Step S6: Calculate the loss between the rendered image and the original image, backpropagate the parameters, and optimize the 3D face model.

[0017] Furthermore, step S1 specifically involves: acquiring and processing several two-dimensional facial images of the same person from three different perspectives, the different perspectives including the frontal face, left side face, and right side face; and using the Dlib algorithm of the facial landmark detector to detect and track 68 facial feature points.

[0018] Furthermore, step S3 specifically includes:

[0019] Step S31: Input the mask image obtained in step S2 into the ResNet network. The input image is fed into the network by concatenating the number of channels, sharing the network weight parameters, and outputting the corresponding 3 feature maps.

[0020] Step S32: Feed the three feature maps into their respective fully connected layers to output pose, camera, and illumination coefficients; on the other hand, stitch and fuse the three feature maps together, and then output the shape and expression parameters of the same person through a fully connected layer.

[0021] Furthermore, the parametric face model is a FLAME parametric model, which consists of two parts: a standard linear blending skin and a blending shape.

[0022] Furthermore, the FLAME parametric model uses a standard mesh model with N vertices, and the mesh vertex coordinates are expressed using the function M(β,θ,ψ):R |β|×|θ|×|ψ| →R 3N Let represent , where β∈R |β| Let θ be the shape parameter, θ∈R |θ| Let ψ be the attitude parameter, ψ∈R |ψ| For facial expression parameters, specifically, the FLAME parameterized model is written in the following form:

[0023] M(β,θ,ψ)=W(T P (β,θ,ψ),J(β),θ,ω);

[0024] Where W(T,J,θ,ω) represents the standard linear skinning function, which applies the template network T along J∈R 3K A transformation θ is performed, and the transformation result is smoothed using skinning weights ω; the position of joint J is related to β, i.e., the shape of the head model; the template network is represented as follows. in The first value represents the average grid, and the next three values ​​represent the shape, expression, and pose, respectively.

[0025] Furthermore, the pose blending function of the FLAME parameterized model also includes a pose coefficient, which can be decomposed into rotation and translation matrices to facilitate the subsequent calculation of relative pose. The relative pose is used in the photometric consistency loss calculation. The face model under the corresponding pose is generated by multiplying the average face model by the rotation matrix and adding the translation matrix.

[0026] Furthermore, step S5 employs a differentiable renderer for texture rendering, specifically as follows:

[0027] I r =R(M,B,c)

[0028] Here, R represents the rendering function, due to the limitations of the low mesh resolution.

[0029] Furthermore, the loss calculation includes: facial landmark loss L lmk Eye closure loss L eye Photometric uniformity loss L photo Face recognition loss L id and regularization loss L reg The details are as follows:

[0030] Facial landmark loss L lmk Calculate the distances between the locations of 68 facial key points and their corresponding points on the surface of the FLAME model.

[0031]

[0032] Eye closure loss L eye Calculate the relative offset between the upper and lower eyelids.

[0033]

[0034] Key points of the upper and lower eyelids are projected onto the FLAME surface M i and M j The offset of the corresponding coordinates; E is the set of upper and lower eyelid marker pairs;

[0035] Photometric uniformity loss L photo :

[0036]

[0037] in, M represents the left mask area of ​​the front view. A→B I represents the mask region rotated from image A to image B. B [u] represents the texture pixels of image B, I A→B [u] represents the texture pixels of image A rotated to image B, and the square of the L2 norm between them is calculated;

[0038] Face recognition loss L id To determine whether the reconstructed faces belong to the same individual, a face recognition network pre-trained on the VGG-Face2 dataset is used for loss calculation. Features of the rendered image and the input image are compared, and the cosine similarity between the two images is calculated.

[0039]

[0040] Regularization loss L reg :

[0041] L reg Regularization: Shape expression and light

[0042] Compared with the prior art, the present invention has the following advantages:

[0043] 1. In this invention, for the self-occlusion problem caused by large-angle facial poses, when matching the parametric model with multi-view facial images, the model parameters of each image are linked with the deformation model. To enhance the geometric constraints between images, photometric consistency is used for constraint, and the final matched 3D facial model is obtained.

[0044] 2. When determining whether the reconstructed faces belong to the same person, this invention uses a face recognition network pre-trained on the VGG-Face2 dataset for loss calculation, compares the features of the rendered image and the input image, and calculates the cosine similarity between the two images, resulting in stronger model robustness. Attached Figure Description

[0045] Figure 1 This is a flowchart of the multi-view face reconstruction process of the present invention.

[0046] Figure 2 This is a schematic diagram of key facial features in one embodiment of the present invention;

[0047] Figure 3 This is a FLAME parameterized model in one embodiment of the present invention. Detailed Implementation

[0048] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0049] Please refer to Figure 1 This invention provides a multi-view face reconstruction method based on a parametric model, comprising the following steps:

[0050] Step S1: Acquire and process several two-dimensional face images of the same person from different perspectives, and then use the Dlib algorithm of the face landmark detector to detect and track facial feature points;

[0051] Step S2: Based on the location of feature points in the 2D face images from different perspectives, mask the unobstructed areas of the frontal face, left side face, and right side face respectively;

[0052] Step S3: Input the mask image obtained in step S2 into the ResNet network to regress the face shape, expression, pose, lighting and camera parameters from different perspectives;

[0053] Step S4: Fit the face shape, expression, pose, lighting and camera parameters obtained in step S3 to the parametric face model to generate the corresponding 3D face model;

[0054] Step S5: Perform texture sampling, and then use a differentiable renderer to generate the rendered image;

[0055] Step S6: Calculate the loss between the rendered image and the original image, backpropagate the parameters, and optimize the 3D face model.

[0056] In this embodiment, the multiple 2D face images are multiple RGB images taken in the same environment. Images taken under different environmental conditions have significant differences in lighting, which can cause color anomalies when reconstructing the face texture map. There is no strict limit to the number of images, but generally three images are required: a frontal face, a left face, and a right face, because these three face images basically completely cover the face area.

[0057] The 68 facial key points mentioned above can constrain the basic shape of the face and locate the approximate position of the face in the image;

[0058] In this embodiment, the ResNet network uses the ResNet50 network structure. The input image is fed into the network by concatenating channels, sharing network weight parameters, and outputting three corresponding feature maps. The three feature maps are then fed into their respective fully connected layers to output pose, camera, and illumination coefficients. On the other hand, the three feature maps are concatenated and fused, and then passed through a fully connected layer to output the shape and expression parameters of the same person.

[0059] The generated 100 shape parameters, 50 expression parameters, 50 texture parameters, 6 pose parameters, 3 camera parameters, and 27 lighting parameters are fitted onto a FLAME parameterized model.

[0060] In this embodiment, preferably, the parametric face model uses the FLAME parametric model, which consists of a standard linear blending skin and a blending shape, enabling the reconstruction of the entire head region, including the face, back of the head, and neck. The standard mesh model used in the FLAME model has a mesh vertex count of N = 5023. The mesh vertex coordinates of the FLAME model can be represented by the function M(β,θ,ψ): R |β|×|θ|×|ψ| →R 3N Let represent , where β∈R |β| Let θ be the shape parameter, θ∈R |θ| Let ψ be the attitude parameter, ψ∈R |ψ| For facial expression parameters.

[0061] The FLAME parameterized model can be written in the following form: M(β,θ,ψ)=W(T) P (β,θ,ψ),J(β),θ,ω), where W(T,J,θ,ω) represents the standard linear skinning function, which applies the template network T along J∈R. 3K A transformation θ is performed, and the transformation result is smoothed using skinning weights ω. The position of joint J is related to β, i.e., the shape of the head model. The template network is represented as follows: in The value represents the average grid, while the others represent shape, expression, and pose, respectively.

[0062] Shape item: Where β = [β1, ..., β β ] T The shape factor; S = [S1, ..., S2] β ]∈R 3N ×|β| The shape basis represents an orthogonal shape, obtained through PCA training.

[0063] Emoticons: in Let ψ be the expression coefficient, and ψ = [ψ1, ..., ψ2]. β ]∈R 3N ×|ψ| This represents an orthogonal emoji base, which is also obtained through training.

[0064] Attitude term: R(θ):R |θ| →R 9K This means that the attitude vector θ∈R 3K+3 Expanding into a matrix and removing the global rotation, we obtain a vector of dimension 9K. The pose term is represented by the formula shown below: Where R n (θ) and R n (θ * ) represent R(θ) and R(θ) respectively. * The nth element of ). P = [P1, ..., P 9K ]∈R 3N×9K The attitude coefficients here include global rotation, rotation of the head around the neck, rotation of the jaw, and rotation of each eyeball.

[0065] An additional pose coefficient is added to the pose mixture function of the FLAME model. This pose coefficient can be decomposed into rotation and translation matrices, facilitating the subsequent calculation of relative pose, which is then used in the photometric consistency loss calculation. The face model in the corresponding pose is generated by multiplying the average face model by the rotation matrix and adding the translation matrix.

[0066] In this embodiment, the appearance model: FLAME has no appearance model, so the PCA space of the Basel Face model is converted to the UV space of FLAME.

[0067] Camera Model: A camera model can project the vertices of a 3D human face onto a 2D plane. This invention uses an orthogonal projection matrix: V = sП(M i )+t, where M represents the vertex, П represents the 2*3 orthogonal projection matrix [[1,0,0],[0,1,0]], s represents the scaling factor, and t represents the translation vector.

[0068] Lighting model: Primarily used to calculate light intensity at a specific point on a face. This invention employs a spherical harmonic model. Where A represents reflectivity, N represents surface normal vector, and B represents shadow texture.

[0069] In this embodiment, texture rendering specifically refers to:

[0070] I r =R(M,B,c)

[0071] Here, R represents the rendering function. Due to the limitation of low mesh resolution, mid-frequency details of the FLAME model are easily lost.

[0072] The loss calculation includes: facial landmark loss (L lmk ), eye closure loss (L) eye ), photometric uniformity loss (L) photo ), face recognition loss (L) id ) and regularization loss (L reg ).

[0073] L loss =L lmk +L eye +L photo +L id +L reg

[0074] Facial landmark loss L lmk Calculate the distances between the locations of 68 facial key points and the corresponding points on the surface of the FLAME model.

[0075]

[0076] Eye closure loss L eye : Calculate the relative offset between the upper and lower eyelids.

[0077] Key points of the upper and lower eyelids are projected onto the FLAME surface M i and M j The offset of the corresponding coordinates. E is the set of upper and lower eyelid marker pairs.

[0078] Photometric uniformity loss L photo To address the self-occlusion problem caused by large-angle facial poses, when matching parametric models with multi-view facial images, the model parameters of each image are linked to the deformation model. To enhance the geometric constraints between images, photometric consistency is used for constraint, and the final matched 3D facial model is obtained.

[0079] M represents the left mask area of ​​the front view. A→B I represents the mask region rotated from image A to image B. B[u] represents the texture pixels of image B, I A→B [u] represents the texture pixels of image A rotated to image B, and the square of the L2 norm between them is calculated.

[0080] Face recognition loss L id To determine whether the reconstructed faces belong to the same person, a face recognition network pre-trained on the VGG-Face2 dataset is used for loss calculation. The features of the rendered image and the input image are compared, and the cosine similarity between the two images is calculated.

[0081]

[0082] Regularization loss L reg To prevent overfitting of shape, expression, and lighting parameters.

[0083] L reg Regularization: Shape expression and light

[0084] The above description is only a preferred embodiment of the present invention. All equivalent changes and modifications made within the scope of the claims of the present invention should be included in the scope of the present invention.

Claims

1. A multi-view face reconstruction method based on a parametric model, characterized in that, Includes the following steps: Step S1: Acquire and process several two-dimensional face images of the same person from different perspectives, and then use the Dlib algorithm of the face landmark detector to detect and track facial feature points; Step S2: Based on the location of feature points in the 2D face images from different perspectives, mask the unobstructed areas of the frontal face, left side face, and right side face respectively; Step S3: Input the mask image obtained in step S2 into the ResNet network to regress the face shape, expression, pose, lighting and camera parameters from different perspectives; Step S4: Fit the face shape, expression, pose, lighting and camera parameters obtained in step S3 to the parametric face model to generate the corresponding 3D face model; Step S5: Perform texture sampling, and then use a differentiable renderer to generate the rendered image; Step S6: Calculate the loss between the rendered image and the original image, backpropagate the parameters, and optimize the 3D face model; The loss calculation includes at least the photometric consistency loss. : in, This indicates the left mask area of ​​the front view. This represents the Mask region rotated from image A to image B. This represents the texture pixels in image B. Represent the texture pixels of image A rotated to image B, and calculate the square of the L2 norm between them; Step S3 specifically involves: Step S31: Input the mask image obtained in step S2 into the ResNet network. The input image is concatenated using the number of channels and fed into the network. The network weight parameters are shared, and the corresponding three feature maps are output. Step S32: Feed the three feature maps into their respective fully connected layers to output pose, camera, and illumination coefficients; on the other hand, stitch and fuse the three feature maps together, and then output the shape and expression parameters of the same person through a fully connected layer.

2. The multi-view face reconstruction method based on a parametric model according to claim 1, characterized in that, Step S1 specifically involves: acquiring and processing several two-dimensional facial images of the same person from three different perspectives, including the frontal face, left side face, and right side face; and using the Dlib algorithm of the facial landmark detector to detect and track 68 facial feature points.

3. The multi-view face reconstruction method based on a parametric model according to claim 1, characterized in that, The parametric face model is a FLAME parametric model, which consists of two parts: a standard linear blending skin and a blending shape.

4. The multi-view face reconstruction method based on a parametric model according to claim 3, characterized in that, The FLAME parametric model uses a standard mesh model with N vertices, and the mesh vertex coordinates are determined by a function. : To indicate, among which For shape parameters, These are attitude parameters. For facial expression parameters, specifically, the FLAME parameterized model is written in the following form: ； in This represents a standard linear skinning function, which will use a template network. along Transform and use skin weights Smooth the transformation results; joints Position and That is, it is related to the shape of the head model; the template network is represented as follows: ,in The first value represents the average grid, and the next three values ​​represent the shape, expression, and pose, respectively.

5. The multi-view face reconstruction method based on a parametric model according to claim 4, characterized in that, The pose blending function of the FLAME parameterized model also includes a pose coefficient, which can be decomposed into a rotation matrix and a translation matrix to facilitate the subsequent solution of the relative pose. The relative pose is used when calculating the photometric consistency loss. The face model under the corresponding pose is generated by multiplying the average face model by the rotation matrix and adding the translation matrix.

6. The multi-view face reconstruction method based on a parametric model according to claim 1, characterized in that, Step S5 employs a differentiable renderer for texture rendering, specifically as follows: Where R represents the rendering function.

7. The multi-view face reconstruction method based on a parametric model according to claim 1, characterized in that, The loss calculation also includes: facial landmark loss. Eye closure loss Facial recognition loss and regularization loss The details are as follows: Facial landmark loss Calculate the distances between the locations of 68 facial key points and their corresponding points on the surface of the FLAME model. Eye closure loss Calculate the relative offset between the upper and lower eyelids. Key points of the upper and lower eyelids are projected onto the FLAME surface. and The offset of the corresponding coordinates; E is the set of upper and lower eyelid marker pairs; Facial recognition loss To determine whether the reconstructed faces belong to the same individual, a face recognition network pre-trained on the VGG-Face2 dataset is used for loss calculation. Features of the rendered image and the input image are compared, and the cosine similarity between the two images is calculated. Regularization loss : Regularization: Shape ,expression and light .

Images