A three-dimensional face reconstruction method, device, electronic device and storage medium
By using lightweight networks and attention mechanisms and graph convolutional neural networks in the three-dimensional face reconstruction method, the problems of high computational cost and insufficient stability of three-dimensional face reconstruction in the existing technology are solved, and efficient and robust three-dimensional face model reconstruction is achieved.
Patent Information
- Application Number
- CN202111534293.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-12-15
- Publication Date
- 2025-03-21
- Estimated Expiration
- 2041-12-15
AI Technical Summary
Existing three-dimensional face reconstruction methods have challenges in recovering accurate three-dimensional face models from two-dimensional images, especially deep learning-based methods have problems with high computational costs, long run time, and insufficient stability and robustness of reconstruction results.
A three-dimensional face reconstruction method based on a single face image is adopted. A lightweight network (such as MobileNet-V1) combined with attention mechanism and graph convolution neural network is used to combine the face parameter recognition model obtained by training, and a three-dimensional vertex coordinate is calculated to generate a complete three-dimensional face model.
It effectively accelerates the speed of model inference, improves the robustness and stability of the reconstructed three-dimensional face model, supports three-dimensional reconstruction of large-pose faces, avoids model distortion and deformation, and the final model size is less than 30M and can run on the CPU.
Smart Images
Figure CN114187407B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer vision technology, and in particular to a three-dimensional face reconstruction method, device, electronic equipment and storage medium. Background Art
[0002] 3D face reconstruction has always been a hot topic in the field of computer vision. 3D face reconstruction can be applied to face recognition, face alignment, face animation, etc. In recent years, although researchers have proposed many methods for 3D face reconstruction, it is still a challenge to restore an accurate 3D face model from a 2D image. In order to simplify the problem, researchers will use 3D face models or templates to perform reconstruction tasks. The most commonly used model is the 3D Morphable Model (3DMM).
[0003] Currently, 3D face reconstruction methods based on 3DMM can be roughly divided into two categories: optimization-based methods and deep learning-based methods.
[0004] The optimization-based method uses a set of 3D face scans to construct a 3D face model that can represent any person, and then optimizes the model by minimizing a series of loss functions to match the input image. The deep learning-based method recovers the 3D face model from the 2D image by learning the mapping relationship between the 2D image and the 3D face model.
[0005] The disadvantage of optimization-based methods is that they need to rely on a large-scale 3D face database to build a face model, and involve a large number of iterative fitting calculations. The common problems of deep learning-based methods are: (1) The introduction of deep convolutional networks (such as VGG16) will increase the computational cost, increase the running time, and require high-performance processors. If the number of layers of the neural network is directly reduced, it is easy to cause the model to not fit well. (2) The stability and robustness of the reconstruction results are insufficient. Specifically, most 3D face reconstruction methods usually do not support face images with large postures (for example, the face yaw angle is up to 90°), which will cause distortion of the reconstructed model. Summary of the invention
[0006] Based on this, the purpose of the present invention is to provide a three-dimensional face reconstruction method, device, electronic device and storage medium, which perform three-dimensional face reconstruction based on a single face image without the need to collect high-precision three-dimensional face data.
[0007] In a first aspect, the present invention provides a 3D face reconstruction method, the method comprising the following steps:
[0008] Acquire a face image to be reconstructed;
[0009] Preprocessing the face image to obtain a face image of standard size;
[0010] Inputting the standard-sized face image into a trained face parameter recognition model to obtain face parameters corresponding to the face image; wherein the face parameters include posture parameters, shape parameters and expression parameters;
[0011] Calculating the three-dimensional vertex coordinates corresponding to the face image according to the face parameters;
[0012] The three-dimensional vertex coordinates are saved to obtain a complete file containing a three-dimensional face model.
[0013] Furthermore, the face parameter recognition model includes a MobileNet-V1 network module, an attention module, a graph convolutional neural network module and a fully connected layer;
[0014] The MobileNet-V1 network module includes a 3*3 standard convolution layer, 13 depth-separable convolution blocks and an average pooling layer connected in sequence; each depth-separable convolution block contains a 3*3 depth-wise convolution layer and a 1*1 point-by-point convolution layer;
[0015] The graph convolutional neural network module includes static graph convolution and dynamic graph convolution.
[0016] Furthermore, the face image of the standard size is input into a trained face parameter recognition model to obtain face parameters corresponding to the face image, including:
[0017] The MobileNet-V1 network module processes the facial image to extract features of the facial image;
[0018] The fully connected layer processes the facial image features to obtain 12-dimensional posture parameters;
[0019] The attention module and the graph convolutional neural network module process the facial image features to obtain 50-dimensional shape parameters and expression parameters.
[0020] Furthermore, the training loss function of the face parameter recognition model includes a fast parameter distance cost function, a vertex distance cost function and a feature point loss function;
[0021] The calculation formula of the fast parameter distance cost function is as follows:
[0022]
[0023] in, is the fast parameter distance cost, T is the matrix obtained by transforming the 12-dimensional posture parameters predicted by the network, α = [α shp , α exp ], which represents the shape parameter α predicted by the network shp and expression parameter α exp , T gt and α gt Represent the transformation matrix and shape and expression parameters from the annotation, w T and w α denote the transformation matrix and the weights of shape and expression parameters respectively;
[0024] The calculation formula of the vertex distance cost function is as follows:
[0025]
[0026] in, is the vertex distance cost, p = [T, α shp , α exp ], which represents all 3DMM parameters predicted by the network, namely the transformation matrix T representing the posture, the shape parameter α shp And the expression parameter α exp , p gt Indicates the marked parameter, V 3d (·) indicates the reconstructed vertices;
[0027] The calculation formula of the feature point loss function is as follows:
[0028]
[0029] in, is the feature point loss, N = 68, indicating 68 facial feature points, q n represents the facial feature points obtained through training, Represents the annotated feature points, obtained by using a face feature point detector.
[0030] Furthermore, the training step of the face parameter recognition model includes:
[0031] Use weighted parameter distance cost And feature point loss Calculating model loss The calculation formula is as follows:
[0032]
[0033] in, is the weight used to balance the two loss magnitudes, m fwpdc and m lmk Corresponding to and Two magnitudes of loss;
[0034] When model training converges, use vertex distance cost And feature point loss Calculating model loss The calculation formula is as follows:
[0035]
[0036] in, With w lmk The same effect is achieved, m vdc and m lmk Corresponding to and Two magnitudes of loss;
[0037] The training is continued until the model converges again, and then the training is stopped to obtain a trained face parameter recognition model.
[0038] Further, according to the face parameters, the three-dimensional vertex coordinates corresponding to the face image are calculated, including:
[0039] Calculate the face shape vector S according to the following formula: model :
[0040]
[0041] in, represents the average face shape, B shp and B exp They represent the shape basis and expression basis obtained by principal component analysis algorithm, α shp and α exp Respectively represent the parameters corresponding to the shape basis and the expression basis;
[0042] According to the following formula, calculate the reconstructed vertex V 3d (p):
[0043]
[0044] Where p = [T, α shp , α exp ], which represents all 3DMM parameters predicted by the network, namely the transformation matrix T representing the posture, the shape parameter α shp And the expression parameter α exp , S model Vector of human face shape.
[0045] Further, the face image is preprocessed to obtain a face image of standard size, including:
[0046] Inputting the face image into a face detector;
[0047] The face detector performs face detection on the face image to obtain a face area in the image;
[0048] Cropping the image using the length of the longer side of the face region frame as the length and width of the cropped image;
[0049] Resize the cropped image to 120*120.
[0050] In a second aspect, the present invention further provides a three-dimensional face reconstruction device, comprising:
[0051] A face image acquisition module, used to acquire the face image to be reconstructed;
[0052] A preprocessing module, used for preprocessing the face image to obtain a face image of standard size;
[0053] A face parameter acquisition module, used for inputting the face image of the standard size into the trained face parameter recognition model to obtain the face parameters corresponding to the face image; wherein the face parameters include posture parameters, shape parameters and expression parameters;
[0054] A three-dimensional vertex coordinate calculation module, used to calculate the three-dimensional vertex coordinates corresponding to the face image according to the face parameters;
[0055] The three-dimensional face model generation module is used to save the three-dimensional vertex coordinates to obtain a complete file containing the three-dimensional face model.
[0056] In a third aspect, the present invention further provides a smart device, comprising:
[0057] at least one memory and at least one processor;
[0058] The memory is used to store one or more programs;
[0059] When the one or more programs are executed by the at least one processor, the at least one processor implements the steps of a three-dimensional face reconstruction method as described in any one of the first aspects of the present invention.
[0060] In a fourth aspect, the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of a three-dimensional face reconstruction method as described in any one of the first aspects of the present invention are implemented.
[0061] The present invention provides a 3D face reconstruction method, device, electronic device and storage medium, which use a lightweight network as the backbone network to effectively speed up the model reasoning speed; the combination of attention mechanism and graph convolutional neural network effectively improves the robustness and stability of the reconstructed 3D face model; adding feature point loss calculation can make the method better support the 3D reconstruction of large-pose faces, so that the reconstructed 3D face model will not be distorted. The size of the trained model finally obtained is less than 30M, and it can support running on the CPU.
[0062] For better understanding and implementation, the present invention is described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0063] Figure 1 A schematic flow chart of a three-dimensional face reconstruction method provided in one embodiment of the present invention;
[0064] Figure 2 A schematic diagram of the structure of a face parameter recognition model used in one embodiment of the present invention;
[0065] Figure 3 A schematic diagram of the training process of a face parameter recognition model used in one embodiment of the present invention;
[0066] Figure 4 The figure is a schematic structural diagram of a three-dimensional face reconstruction device provided in one embodiment of the present invention. DETAILED DESCRIPTION
[0067] In order to make the objectives, technical solutions and advantages of the present application more clear, the embodiments of the present application will be further described in detail below with reference to the accompanying drawings.
[0068] It should be clear that the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the embodiments of the present application.
[0069] The terms used in the embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to limit the embodiments of the present application. The singular forms of "a", "said" and "the" used in the embodiments of the present application and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.
[0070] When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. On the contrary, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the attached claims. In the description of the present application, it should be understood that the terms "first", "second", "third", etc. are only used to distinguish similar objects, and do not have to be used to describe a specific order or sequence, nor can they be understood as indicating or implying relative importance. For those of ordinary skill in the art, the specific meanings of the above terms in the present application can be understood according to the specific circumstances.
[0071] In addition, in the description of this application, unless otherwise specified, "plurality" means two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects are in an "or" relationship.
[0072] In a specific embodiment, the present invention provides a 3D face reconstruction method, using 3D face synthesis data based on a 3D Morphable Model (3DMM). The 3D face synthesis data is specifically the 3DMM parameters of the face, and each 3DMM sample contains 199*1 shape parameters, 199*1 texture parameters, 1*7 color parameters, 29*1 expression parameters, 1*10 illumination parameters, 1*7 posture parameters, 21 2D face feature points, 68 3D face feature points and rotation angles. The data set used in the present invention has 687,854 samples after being expanded by Guo Jianzhu et al., and the 3DMM parameters are reduced, retaining only the first 40-dimensional shape parameters, 10-dimensional expression parameters and 12-dimensional 3*4 posture transformation matrices calculated by posture parameters and rotation angles.
[0073] In view of the problems in the background technology, the present application provides a 3D face reconstruction method, such as Figure 1 As shown, the method comprises the following steps:
[0074] S01: Obtain a face image to be reconstructed.
[0075] S02: Preprocessing the face image to obtain a face image of standard size.
[0076] Specifically, the preprocessing includes the following sub-steps:
[0077] S021: Input the face image into a face detector.
[0078] S022: The face detector performs face detection on the face image to obtain a face area in the image.
[0079] In a preferred embodiment, the face detector uses a multi-task convolutional neural network (MTCNN) to perform face detection.
[0080] S023: Cropping the image by taking the length of the longer side of the face region frame as the length and width of the cropped image.
[0081] S024: Scale the cropped image to 120*120.
[0082] S03: Inputting the standard-sized face image into a trained face parameter recognition model to obtain face parameters corresponding to the face image; wherein the face parameters include posture parameters, shape parameters and expression parameters.
[0083] Preferably, Figure 2 As shown, the face parameter recognition model includes a MobileNet-V1 network module, an attention module, a graph convolutional neural network module and a fully connected layer. The MobileNet-V1 network module includes a 3*3 standard convolution layer, 13 depth-separable convolution blocks and an average pooling layer connected in sequence; each depth-separable convolution block contains a 3*3 depth convolution layer and a 1*1 point-by-point convolution layer. The graph convolutional neural network module includes static graph convolution and dynamic graph convolution.
[0084] Based on the face parameter recognition model, obtaining the face parameters corresponding to the face image includes the following sub-steps:
[0085] S031: The MobileNet-V1 network module processes the facial image and extracts features of the facial image.
[0086] S032: The fully connected layer processes the facial image features to obtain 12-dimensional posture parameters.
[0087] S033: The attention module and the graph convolutional neural network module process the facial image features to obtain 50-dimensional shape parameters and expression parameters.
[0088] S04: Calculate and obtain the three-dimensional vertex coordinates corresponding to the face image according to the face parameters.
[0089] In a preferred embodiment, the calculation method includes:
[0090] S041: Calculate the face shape vector S according to the following formula model :
[0091]
[0092] in, represents the average face shape, B shp and B exp They represent the shape basis and expression basis obtained by principal component analysis algorithm, α shp and α exp Respectively represent the corresponding parameters of the shape basis and the expression basis.
[0093] S042: Calculate the reconstructed vertex V according to the following formula 3d (p):
[0094]
[0095] Where p = [T, α shp , α exp ], which represents all 3DMM parameters predicted by the network, namely the transformation matrix T representing the posture, the shape parameter α shp And the expression parameter α exp , S model Vector of human face shape.
[0096] S05: Save the three-dimensional vertex coordinates to obtain a complete file containing a three-dimensional face model.
[0097] In a specific embodiment, according to the format of the OBJ file, the vertex coordinates are written into the OBJ file, and the triangular face data file is read and the triangular face data is written into the same OBJ file, and finally a complete OBJ format file containing a three-dimensional face model is obtained. The triangular face data file is open source data provided by Guo Jianzhu (Guo J, Zhu X, Yang Y, et al. Towards Fast, Accurate and Stable 3D Dense Face Alignment [C]. European Conference on Computer Vision (ECCV), 2020.). The OBJ format file can be opened by software such as MeshLab to obtain a visualized three-dimensional face model.
[0098] In a preferred embodiment, a 3D face reconstruction method provided by the present invention also includes data preparation and model training process of a face parameter recognition model, which specifically includes the following steps.
[0099] S11: Data sample acquisition.
[0100] The original data sample set comes from 300W-LP, which contains two-dimensional face images and three-dimensional face synthesis data corresponding to each image. The three-dimensional face synthesis data is specifically the 3DMM parameters of the face. Each 3DMM sample contains 199*1 shape parameters, 199*1 texture parameters, 1*7 color parameters, 29*1 expression parameters, 1*10 illumination parameters, 1*7 posture parameters, 21 two-dimensional face feature points, 68 three-dimensional face feature points and rotation angles. After being expanded by Guo Jianzhu et al., the data set has 687,854 samples, and the 3DMM parameters are reduced, retaining only the first 40-dimensional shape parameters, 10-dimensional expression parameters and 12-dimensional 3*4 posture transformation matrices calculated by posture parameters and rotation angles.
[0101] S12: Feature point extraction.
[0102] Use the trained facial feature point detector to detect all the two-dimensional face images in the sample and extract 68 two-dimensional facial feature points of each image, and then record the extracted data in a text file. The facial feature point detector is the Face Alignment Network (FAN). The face successfully detected by FAN returns the 68 two-dimensional facial feature points of the corresponding face. If the detection fails, it prompts the failure and returns the file name corresponding to the image. Then delete the corresponding image and the corresponding three-dimensional data according to the image file name that failed the detection. Finally, 626,088 training set samples and 50,807 validation set samples were obtained.
[0103] S13: Network framework design. The network framework is as follows Figure 2 shown.
[0104] The method is based on 3DMM, and the original formula is:
[0105]
[0106] Among them, S model and T model Represents the face shape vector and texture vector, and represent the average face shape and average facial texture, respectively, and B shp and B tex Respectively represent the shape and texture basis obtained by the principal component analysis algorithm, α shp and α tex Respectively represent the corresponding parameters. In the present invention, in order to ensure the rapid reconstruction of the 3D face model, the texture vector is not included in the calculation. Therefore, the modified 3DMM formula is:
[0107]
[0108] Among them, the increase in B exp is the expression base obtained through principal component analysis, α exp is the corresponding expression parameter. Given a 2D face image, the network provided by this method regresses a 62-dimensional vector. This vector represents the 3DMM parameters, of which 12 dimensions are 3*4 matrices to represent face pose data, 40 dimensions are shape parameters, and 10 dimensions are expression parameters.
[0109] The network first uses a lightweight network as a skeleton to extract facial image features. The role of the lightweight network is to reduce the model size while maintaining model performance and improve the speed of model inference.
[0110] The lightweight network used as the skeleton is a modified MobileNet-V1. The network structure first consists of a 3*3 standard convolution layer, followed by 13 depth-wise separable convolution blocks, each of which contains a 3*3 depth-wise convolution layer and a 1*1 point-wise convolution layer, and finally an average pooling layer, for a total of 27 layers.
[0111] Then different parameters are regressed in two parts: one part obtains 12-dimensional posture parameters through a fully connected layer, and the other part obtains 50-dimensional shape parameters and expression parameters through a network composed of an attention mechanism and a graph convolutional neural network.
[0112] The role of the attention mechanism is to focus on the semantic information of each 3DMM parameter in the features extracted by the lightweight network in order to instruct the network to selectively aggregate. First, the network automatically learns to obtain the attention mask M based on the input face features, which captures the relationship between the input features and the corresponding 3DMM parameters. Further, the attention mask M is used to construct the matrix X representing the contextual relationship c Specifically, for each 3DMM parameter, X c It is the weighted sum of the input feature X and its corresponding attention mask M. The specific formula is as follows:
[0113]
[0114] in, represents the weight of the kth attention mask, x ij represents the feature vector of feature map X at (i, j), where h and w are the height and width of the input image respectively.
[0115] The graph convolutional neural network includes static graph convolution and dynamic graph convolution, which helps to restore the geometric details of the three-dimensional face shape. The expression formula of static graph convolution is as follows:
[0116] Hs =σ(A s X c W s ) (4)
[0117] Among them, H s represents the updated node, A s is the adjacency matrix of the static graph, X c is the output obtained through the attention mechanism mentioned above, W s is the weight matrix of the static graph, and σ(·) refers to the activation function LeakyReLU. s and W s is obtained randomly through convolution. Since the adjacency matrix A in the static graph convolution s It is fixed, and it is unreasonable to use the same adjacency matrix for all inputs, so dynamic graph convolution is introduced. In dynamic graph convolution, the adjacency matrix is dynamically updated. Recalculating the adjacency matrix of each input can better propagate information between similar structures and accelerate the learning of local semantic information. Dynamic graph convolution can be defined as:
[0118] H d =σ(A d H s W d ) (5)
[0119] Among them, H d H is the output 50-dimensional shape and expression parameters, s From formula (4), σ(·) refers to the activation function LeakyReLU, W d Update matrix for the state of dynamic graph convolution, adjacency matrix A d The specific definition is:
[0120] A d =δ(W a H′ s ) (6)
[0121] Among them, δ(·) is the Sigmoid activation function, W a is the weight matrix obtained after convolution, H′ s It is made by H s Constructed by concatenating it with its global expression.
[0122] S14: Loss function design.
[0123] There are three main loss functions: the first is fast weighted parameter distance cost calculation, which mainly calculates the error loss between the parameters obtained by network training and the actual annotated parameters; the second is vertex distance cost calculation, which mainly calculates the error loss between the vertices of the three-dimensional model reconstructed by the 3DMM parameters obtained through training and the annotated vertices; the third is feature point loss calculation, which mainly calculates the error loss between the 68 facial feature points obtained through training and the feature points extracted from S12. The purpose of adding feature point loss calculation is to improve the accuracy of face alignment so that this method can better support the reconstruction of large-pose face images.
[0124] The fast parameter distance cost calculation formula is as follows:
[0125]
[0126] in, is the fast parameter distance cost, T is the matrix obtained by transforming the 12-dimensional posture parameters predicted by the network, α = [α shp , α exp ], which represents the shape parameter α predicted by the network shp , and the expression parameter α exp , T gt and α gt Represent the transformation matrix and shape and expression parameters from the annotation, w T and w α Represent the transformation matrix and the weights of shape and expression parameters respectively.
[0127] The vertex distance cost calculation formula is as follows:
[0128]
[0129] in, is the vertex distance cost, p = [T, α shp , α exp ], which represents all 3DMM parameters predicted by the network, namely the transformation matrix T representing the posture, the shape parameter α shp And the expression parameter α exp , p gt Indicates the marked parameter, V 3d (·) is the reconstructed vertex, which is calculated as:
[0130]
[0131] Where p and T have the same meaning as p and T in formula (8), p is all 3DMM parameters, T is a transformation matrix, S model From formula (2).
[0132] The feature point loss The calculation formula is as follows:
[0133]
[0134] in, is the feature point loss, N=68 means 68 facial feature points, q n Represents the facial feature points obtained through training. The specific method of obtaining the facial feature points is to reproject the three-dimensional facial feature points reconstructed by the training parameters onto the two-dimensional image. represents the feature points obtained in S12.
[0135] S15: Model training.
[0136] The two-dimensional face image passed through S12 is input into the network designed in S13 for training. First, the weighted parameter distance cost is used And feature point loss Calculating model loss The specific calculation formula is as follows:
[0137]
[0138] in is a weight used to balance the two loss magnitudes, m fwpdc and m lmk Corresponding to and Two orders of magnitude of loss.
[0139] When model training converges, use vertex distance cost And feature point loss Calculating model loss The calculation formula is as follows:
[0140]
[0141] in, With w lmk The same effect is achieved, m vdc and m lmk Corresponding to and The magnitude of the two losses. Then train until the model converges again, stop training, and finally get the trained network.
[0142] The present application also provides a three-dimensional face reconstruction device. Figure 4 As shown, the 3D face reconstruction device 400 includes:
[0143] A face image acquisition module 401 is used to acquire a face image to be reconstructed;
[0144] A preprocessing module 402 is used to preprocess the face image to obtain a face image of standard size;
[0145] The face parameter acquisition module 403 is used to input the face image of the standard size into the trained face parameter recognition model to obtain the face parameters corresponding to the face image; wherein the face parameters include posture parameters, shape parameters and expression parameters;
[0146] A three-dimensional vertex coordinate calculation module 404 is used to calculate the three-dimensional vertex coordinates corresponding to the face image according to the face parameters;
[0147] The 3D face model generation module 405 is used to save the 3D vertex coordinates to obtain a complete file containing the 3D face model.
[0148] Preferably, the face parameter recognition model includes a MobileNet-V1 network module, an attention module, a graph convolutional neural network module and a fully connected layer;
[0149] The MobileNet-V1 network module includes a 3*3 standard convolution layer, 13 depth-separable convolution blocks and an average pooling layer connected in sequence; each depth-separable convolution block contains a 3*3 depth-wise convolution layer and a 1*1 point-by-point convolution layer;
[0150] The graph convolutional neural network module includes static graph convolution and dynamic graph convolution.
[0151] Preferably, the face parameter acquisition module includes:
[0152] A feature extraction unit, configured to enable the MobileNet-V1 network module to process the face image and extract features of the face image;
[0153] A posture parameter acquisition unit, used to enable the fully connected layer to process the facial image features to obtain 12-dimensional posture parameters;
[0154] The shape parameter and expression parameter acquisition unit is used to enable the attention module and the graph convolutional neural network module to process the facial image features to obtain 50-dimensional shape parameters and expression parameters.
[0155] Preferably, the training loss function of the face parameter recognition model includes a fast parameter distance cost function, a vertex distance cost function and a feature point loss function;
[0156] The calculation formula of the fast parameter distance cost function is as follows:
[0157]
[0158] in, is the fast parameter distance cost, T is the matrix obtained by transforming the 12-dimensional posture parameters predicted by the network, α = [α shp , α exp ], which represents the shape parameter α predicted by the network shp , and the expression parameter α exp , T gt and α gt Represent the transformation matrix and shape and expression parameters from the annotation, w T and w α denote the transformation matrix and the weights of shape and expression parameters respectively;
[0159] The calculation formula of the vertex distance cost function is as follows:
[0160]
[0161] in, is the vertex distance cost, p = [T, α shp , α exp ], which represents all 3DMM parameters predicted by the network, namely the transformation matrix T representing the posture, the shape parameter α shp And the expression parameter α exp , p gt Indicates the marked parameter, V 3d (·) indicates the reconstructed vertices;
[0162] The calculation formula of the feature point loss function is as follows:
[0163]
[0164] in, is the feature point loss, N = 68, indicating 68 facial feature points, q n represents the facial feature points obtained through training, Represents the annotated feature points, obtained by using a face feature point detector.
[0165] Preferably, the training step of the face parameter recognition model includes:
[0166] Use weighted parameter distance cost And feature point loss Calculating model loss The calculation formula is as follows:
[0167]
[0168] in, is the weight used to balance the two loss magnitudes, m fwpdc and mlmk Corresponding to and Two magnitudes of loss;
[0169] When model training converges, use vertex distance cost And feature point loss Calculating model loss The calculation formula is as follows:
[0170]
[0171] in, With w lmk The same effect is achieved, m vdc and m lmk Corresponding to and Two magnitudes of loss;
[0172] The training is continued until the model converges again, and then the training is stopped to obtain a trained face parameter recognition model.
[0173] Preferably, the three-dimensional vertex coordinate calculation module includes:
[0174] The face shape vector calculation unit is used to calculate the face shape vector S according to the following formula: model :
[0175]
[0176] in, represents the average face shape, B shp and B exp They represent the shape basis and expression basis obtained by principal component analysis algorithm, α shp and α exp Respectively represent the parameters corresponding to the shape basis and the expression basis;
[0177] The vertex calculation unit is used to calculate the reconstructed vertex V according to the following formula: 3d (p):
[0178]
[0179] Where p = [T, α shp , α exp ], which represents all 3DMM parameters predicted by the network, namely the transformation matrix T representing the posture, the shape parameter α shp And the expression parameter α exp , S model Vector of human face shape.
[0180] Preferably, the preprocessing module comprises:
[0181] An image transmission unit, used for inputting the face image into a face detector;
[0182] A face detection unit, configured to enable the face detector to perform face detection on the face image to obtain a face region in the image;
[0183] A cropping unit, used to crop the image by taking the length of the longer side of the face region frame as the length and width of the cropped image;
[0184] The image scaling unit is used to scale the cropped image to 120*120.
[0185] The present application also provides a smart device, including:
[0186] at least one memory and at least one processor;
[0187] The memory is used to store one or more programs;
[0188] When the one or more programs are executed by the at least one processor, the at least one processor implements the steps of the three-dimensional face reconstruction method as described above.
[0189] The present application also provides a computer-readable storage medium.
[0190] The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the three-dimensional face reconstruction method as described above are implemented.
[0191] The present invention provides a 3D face reconstruction method, device, electronic device and storage medium, which use a lightweight network as the backbone network to effectively speed up the model reasoning speed; the combination of attention mechanism and graph convolutional neural network effectively improves the robustness and stability of the reconstructed 3D face model; adding feature point loss calculation can make the method better support the 3D reconstruction of large-pose faces, so that the reconstructed 3D face model will not be distorted. The size of the trained model finally obtained is less than 30M, and it can support running on the CPU.
[0192] The above-mentioned embodiments only express several implementation methods of the present invention, and the description is relatively specific and detailed, but it cannot be understood as limiting the scope of the invention patent. It should be pointed out that for ordinary technicians in this field, several modifications and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention.
Claims
1. A three-dimensional face reconstruction method, characterized in that: The following steps are involved: Acquire a face image to be reconstructed; Preprocessing the face image to obtain a face image of standard size; Input the standard-sized face image into a trained face parameter recognition model to obtain face parameters corresponding to the face image; wherein the face parameters include posture parameters, shape parameters and expression parameters; the face parameter recognition model includes a MobileNet-V1 network module, an attention module, a graph convolutional neural network module and a fully connected layer; The MobileNet-V1 network module includes a 3*3 standard convolution layer, 13 depth-separable convolution blocks and an average pooling layer connected in sequence; each depth-separable convolution block contains a 3*3 depth convolution layer and a 1*1 point-by-point convolution layer; the graph convolution neural network module includes static graph convolution and dynamic graph convolution; The MobileNet-V1 network module processes the facial image to extract features of the facial image; The fully connected layer processes the facial image features to obtain 12-dimensional posture parameters; The attention module and the graph convolutional neural network module process the facial image features to obtain 50-dimensional shape parameters and expression parameters; The training loss function of the face parameter recognition model includes a fast parameter distance cost function, a vertex distance cost function and a feature point loss function; The calculation formula of the fast parameter distance cost function is as follows: in, is the fast parameter distance cost, T is the matrix obtained by transforming the 12-dimensional posture parameters predicted by the network, α = [α shp, α exp ], which represents the shape parameter α predicted by the network shp and expression parameter α exp , T gt and α gt Represent the transformation matrix and shape and expression parameters from the annotation, w T and w α denote the transformation matrix and the weights of shape and expression parameters respectively; The calculation formula of the vertex distance cost function is as follows: in, is the vertex distance cost, p = [T, α shp, α exp ], which represents all 3DMM parameters predicted by the network, namely the transformation matrix T representing the posture, the shape parameter α shp And the expression parameter α exp , p gt Indicates the marked parameter, V 3d (·) indicates the reconstructed vertices; The calculation formula of the feature point loss function is as follows: in, is the feature point loss, N = 68, indicating 68 facial feature points, q n represents the facial feature points obtained through training, Represents the annotated feature points, obtained by using a face feature point detector; The training steps of the face parameter recognition model include: Use weighted parameter distance cost And feature point loss Calculating model loss The calculation formula is as follows: in, is the weight used to balance the two loss magnitudes, m fwpdc and m lmk Corresponding to and Two magnitudes of loss; When model training converges, use vertex distance cost And feature point loss Calculating model loss The calculation formula is as follows: in, With w lmk The same effect is achieved, m vdc and m lmk Corresponding to and Two magnitudes of loss; The training is continued until the model converges again, and then the training is stopped to obtain a trained face parameter recognition model; Calculating the three-dimensional vertex coordinates corresponding to the face image according to the face parameters includes: According to the following formula, calculate the face shape vector S model : in, represents the average face shape, B shp and B exp They represent the shape basis and expression basis obtained by principal component analysis algorithm, α shp and α exp Respectively represent the parameters corresponding to the shape basis and the expression basis; According to the following formula, calculate the reconstructed vertex V 3d (p): Where p = [T, α shp, α exp ], which represents all 3DMM parameters predicted by the network, namely the transformation matrix T representing the posture, the shape parameter α shp And the expression parameter α exp , S model is the face shape vector; The three-dimensional vertex coordinates are saved to obtain a complete file containing a three-dimensional face model.
2. A three-dimensional face reconstruction method according to claim 1, characterized in that: Preprocessing the face image to obtain a face image of standard size includes: Inputting the face image into a face detector; The face detector performs face detection on the face image to obtain a face area in the image; Cropping the image using the length of the longer side of the face region frame as the length and width of the cropped image; Resize the cropped image to 120*120.
3. A three-dimensional face reconstruction device, characterized in that: include: A face image acquisition module, used to acquire the face image to be reconstructed; A preprocessing module, used for preprocessing the face image to obtain a face image of standard size; A face parameter acquisition module, used to input the face image of standard size into a trained face parameter recognition model to obtain face parameters corresponding to the face image; wherein the face parameters include posture parameters, shape parameters and expression parameters; the face parameter recognition model includes a MobileNet-V1 network module, an attention module, a graph convolutional neural network module and a fully connected layer; The MobileNet-V1 network module includes a 3*3 standard convolution layer, 13 depth-separable convolution blocks and an average pooling layer connected in sequence; each depth-separable convolution block contains a 3*3 depth convolution layer and a 1*1 point-by-point convolution layer; the graph convolution neural network module includes static graph convolution and dynamic graph convolution; The MobileNet-V1 network module processes the facial image to extract features of the facial image; The fully connected layer processes the facial image features to obtain 12-dimensional posture parameters; The attention module and the graph convolutional neural network module process the facial image features to obtain 50-dimensional shape parameters and expression parameters; The training loss function of the face parameter recognition model includes a fast parameter distance cost function, a vertex distance cost function and a feature point loss function; The calculation formula of the fast parameter distance cost function is as follows: in, is the fast parameter distance cost, T is the matrix obtained by transforming the 12-dimensional posture parameters predicted by the network, α = [α shp, α exp ], which represents the shape parameter α predicted by the network shp and expression parameter α exp , T gt and α gt Represent the transformation matrix and shape and expression parameters from the annotation, w T and w α denote the transformation matrix and the weights of shape and expression parameters respectively; The calculation formula of the vertex distance cost function is as follows: in, is the vertex distance cost, p = [T, α shp, α exp ], which represents all 3DMM parameters predicted by the network, namely the transformation matrix T representing the posture, the shape parameter α shp And the expression parameter α exp , p gt Indicates the marked parameter, V 3d (·) indicates the reconstructed vertices; The calculation formula of the feature point loss function is as follows: in, is the feature point loss, N = 68, indicating 68 facial feature points, q n represents the facial feature points obtained through training, Represents the annotated feature points, obtained by using a face feature point detector; The training steps of the face parameter recognition model include: Use weighted parameter distance cost And feature point loss Calculating model loss The calculation formula is as follows: in, is the weight used to balance the two loss magnitudes, m fwpdc and m lmk Corresponding to and Two magnitudes of loss; When model training converges, use vertex distance cost And feature point loss Calculating model loss The calculation formula is as follows: in, With w lmk The same effect is achieved, m vdc and m lmk Corresponding to and Two magnitudes of loss; The training is continued until the model converges again, and then the training is stopped to obtain a trained face parameter recognition model; A three-dimensional vertex coordinate calculation module is used to calculate the three-dimensional vertex coordinates corresponding to the face image according to the face parameters; wherein the three-dimensional vertex coordinate calculation module includes executing the following sub-steps: According to the following formula, calculate the face shape vector S model : in, represents the average face shape, B shp and B exp They represent the shape basis and expression basis obtained by principal component analysis algorithm, α shp and α exp Respectively represent the parameters corresponding to the shape basis and the expression basis; According to the following formula, calculate the reconstructed vertex V 3d (p): Where p = [T, α shp, α exp ], which represents all 3DMM parameters predicted by the network, namely the transformation matrix T representing the posture, the shape parameter α shp And the expression parameter α exp , S model is the face shape vector; The three-dimensional face model generation module is used to save the three-dimensional vertex coordinates to obtain a complete file containing the three-dimensional face model.
4. A smart device, characterized in that: include: at least one memory and at least one processor; The memory is used to store one or more programs; When the one or more programs are executed by the at least one processor, the at least one processor implements the steps of a three-dimensional face reconstruction method as described in any one of claims 1-2.
5. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of a three-dimensional face reconstruction method as described in any one of claims 1-2 are implemented.