Three-dimensional face model reconstruction method, device, storage medium and computer equipment

Abstract

The present application discloses a method, device, storage medium and computer equipment for reconstructing a three-dimensional face model. The method includes: inputting a target two-dimensional image into a trained face reconstruction model, outputting a corresponding three-dimensional face model, wherein the face reconstruction model is generated based on cost function training, wherein the cost function includes a first cost function, a second cost function and a third cost function, wherein the first cost function is obtained based on a predicted three-dimensional face model corresponding to a sample two-dimensional image and a standard three-dimensional face model corresponding to the sample two-dimensional image, the second cost function is obtained based on a predicted three-dimensional face model and a smoothed three-dimensional face model after smoothing the predicted three-dimensional face model, and the third cost function is obtained based on facial key points after two-dimensional projection of the predicted three-dimensional face model and facial key points in the sample two-dimensional image. Using the embodiment of the present application, a three-dimensional face model reconstructed using a pre-trained face reconstruction model has an outstanding smoothing effect.

Description

Technical Field

[0001] The present application relates to the technical field of computer equipment application, and in particular to a three-dimensional face model reconstruction method, device, storage medium and computer equipment. Background Art

[0002] 3D face model reconstruction refers to the reconstruction of a 3D face model from one or more 2D face images. Compared with 2D face images, 3D face reconstruction has one more dimension and is widely used in movies, games and other fields. 3D face reconstruction technology can restore the 3D shape of the face in 2D face images, and has high application value in animation production, online games and other fields, and has broad application prospects. Summary of the invention

[0003] The embodiment of the present application provides a method, device, storage medium and computer equipment for reconstructing a 3D face model, wherein the 3D face model reconstructed using a pre-trained face reconstruction model has an outstanding smoothing effect. The technical solution is as follows:

[0004] In a first aspect, an embodiment of the present application provides a method for reconstructing a three-dimensional face model, characterized in that the method comprises:

[0005] Acquire a target two-dimensional image;

[0006] Inputting the target two-dimensional image into a trained face reconstruction model, and outputting a three-dimensional face model corresponding to the target two-dimensional image;

[0007] Wherein, the face reconstruction model is generated based on cost function training, and the cost function includes a first cost function, a second cost function and a third cost function, the first cost function is obtained based on a predicted three-dimensional face model corresponding to a sample two-dimensional image and a standard three-dimensional face model corresponding to the sample two-dimensional image, the second cost function is obtained based on the predicted three-dimensional face model and a smoothed three-dimensional face model after smoothing the predicted three-dimensional face model, and the third cost function is obtained based on facial key points after two-dimensional projection of the predicted three-dimensional face model and facial key points in the sample two-dimensional image;

[0008] The predicted three-dimensional face model is obtained by predicting the sample two-dimensional image based on the created initial face reconstruction model, and the standard three-dimensional face model is obtained by constraining and reconstructing the average face three-dimensional model based on the face key points in the sample two-dimensional image.

[0009] In a second aspect, an embodiment of the present application provides a 3D face model reconstruction device, the 3D face model reconstruction device comprising:

[0010] An image acquisition module, used for acquiring a target two-dimensional image;

[0011] A model prediction module, used to input the target two-dimensional image into a trained face reconstruction model, and output a three-dimensional face model corresponding to the target two-dimensional image;

[0012] Wherein, the face reconstruction model is generated based on cost function training, and the cost function includes a first cost function, a second cost function and a third cost function, the first cost function is obtained based on a predicted three-dimensional face model corresponding to a sample two-dimensional image and a standard three-dimensional face model corresponding to the sample two-dimensional image, the second cost function is obtained based on the predicted three-dimensional face model and a smoothed three-dimensional face model after smoothing the predicted three-dimensional face model, and the third cost function is obtained based on facial key points after two-dimensional projection of the predicted three-dimensional face model and facial key points in the sample two-dimensional image;

[0013] The predicted three-dimensional face model is obtained by predicting the sample two-dimensional image based on the created initial face reconstruction model, and the standard three-dimensional face model is obtained by constraining and reconstructing the average face three-dimensional model based on the face key points in the sample two-dimensional image.

[0014] In a third aspect, an embodiment of the present application provides a storage medium, wherein the storage medium stores at least one instruction, and the at least one instruction is suitable for being loaded by a processor and executing the above-mentioned method steps.

[0015] In a fourth aspect, an embodiment of the present application provides a computer device, which may include: a processor and a memory; wherein the memory stores at least one instruction, and the at least one instruction is suitable for being loaded by the processor and executing the above-mentioned method steps.

[0016] The beneficial effects brought about by the technical solutions provided by some embodiments of the present application include at least:

[0017] The three-dimensional face model reconstruction method provided in the embodiment of the present application is adopted, the acquired target two-dimensional image is input into the face reconstruction model generated by training based on the cost function, and then the three-dimensional face model corresponding to the target two-dimensional image is output, the face reconstruction model is trained and generated using a first cost function constructed based on the predicted three-dimensional face model corresponding to the sample two-dimensional image and the standard three-dimensional face model corresponding to the sample two-dimensional image, a second cost function constructed based on the predicted three-dimensional face model and the smoothed three-dimensional face model after the predicted three-dimensional face model is smoothed, and a third cost function constructed based on the face key points after the predicted three-dimensional face model is two-dimensionally projected and the face key points in the sample two-dimensional image. Since the face reconstruction model is smoothed during the training process, the three-dimensional face model reconstructed using the face reconstruction model has an outstanding smoothing effect, and a good smoothing effect can be achieved without adding additional smoothing calculations during the application of the face reconstruction model. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 A schematic diagram of a process for reconstructing a three-dimensional face model provided in an embodiment of the present application;

[0020] Figure 2 An example schematic diagram of a three-dimensional face reconstruction provided in an embodiment of the present application;

[0021] Figure 3 A schematic diagram of a process for reconstructing a three-dimensional face model provided in an embodiment of the present application;

[0022] Figure 4 A schematic diagram of a process for reconstructing a three-dimensional face model provided in an embodiment of the present application;

[0023] Figure 5 A flowchart of reconstructing a standard three-dimensional face model provided in an embodiment of the present application;

[0024] Figure 6 A flowchart of a training face reconstruction model provided in an embodiment of the present application;

[0025] Figure 7 A schematic diagram of a process for reconstructing a three-dimensional face model provided in an embodiment of the present application;

[0026] Figure 8 A schematic diagram of a process for reconstructing a three-dimensional face model provided in an embodiment of the present application;

[0027] Fig. 9 An example schematic diagram of a stylized processing provided in an embodiment of the present application;

[0028] Fig.10 An example schematic diagram of reconstructing a three-dimensional face model using facial key point constraints provided in an embodiment of the present application;

[0029] Fig.11 A schematic diagram of the structure of a three-dimensional face model reconstruction device provided in an embodiment of the present application;

[0030] Fig.12 A schematic diagram of the structure of a three-dimensional face model reconstruction device provided in an embodiment of the present application;

[0031] Fig.13 A schematic diagram of the structure of a model training module provided in an embodiment of the present application;

[0032] Fig.14 A schematic diagram of the structure of a cost function construction unit provided in an embodiment of the present application;

[0033] Fig.15 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0034] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0035] In the description of the present application, it should be understood that the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. In the description of the present application, it should be noted that, unless otherwise clearly specified and limited, "including" and "having" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally also includes steps or units that are not listed, or optionally also includes other steps or units inherent to these processes, methods, products or devices. For those of ordinary skill in the art, the specific meanings of the above terms in the present application can be understood in specific circumstances. In addition, in the description of the present application, unless otherwise specified, "multiple" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B, which can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects before and after are an "or" relationship.

[0036] In order to more clearly describe the technical solution of the embodiment of the present invention, some concepts in the present invention are described in detail before the description to facilitate a better understanding of the solution.

[0037] Cost function: Usually, in order to obtain the parameters for training a logistic regression model, a cost function is required, and the parameters are obtained by training the cost function.

[0038] Stylization: transforming a two-dimensional image from one style domain to another, such as converting a realistic two-dimensional image to a cartoon two-dimensional image.

[0039] Key points of the face: important feature points such as eyes, nose tip, mouth corners, eyebrows, and contour points of various facial parts.

[0040] Traditional 3D face reconstruction methods are mostly based on image information, such as image brightness, edge information, linear perspective, color, relative height, parallax, etc., to perform 3D face reconstruction. Model-based 3D face reconstruction methods are currently more popular 3D face reconstruction methods; popular models include the universal face model (CANDIDE-3) and the 3D deformation model (3D MM) and its variant models. The 3D face reconstruction algorithms based on them include both traditional algorithms and deep learning algorithms.

[0041] In the prior art, whether it is three-dimensional face reconstruction based on traditional algorithms or three-dimensional face reconstruction based on deep learning algorithms, most of them use smoothing to smooth the reconstructed three-dimensional face model, that is, smoothing is performed after the three-dimensional face model is reconstructed. However, this smoothing method lacks image prior feature information and cannot achieve a good smoothing effect on the face model, resulting in some areas with heavier defects still having defects.

[0042] The present application proposes a method for reconstructing a three-dimensional face model based on a deep learning algorithm. By training a face reconstruction model based on deep learning, three-dimensional face reconstruction is performed on a two-dimensional image based on the face reconstruction model. Smoothness training is added in the process of training the face reconstruction model based on deep learning, so that the trained face reconstruction model has a better smoothing effect when reconstructing a three-dimensional face model.

[0043] The following is a detailed description in conjunction with specific embodiments. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the attached claims. The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily follow the steps shown. For example, some steps are parallel and do not have a strict logical order, so the actual execution order is variable.

[0044] See also Figure 1 , is a flow chart of a method for reconstructing a three-dimensional face model provided in an embodiment of the present application. Figure 1 As shown, the three-dimensional face model reconstruction method may include the following steps S101 to S103.

[0045] S101, acquiring a target two-dimensional image;

[0046] The target two-dimensional image refers to a two-dimensional face image that needs to be reconstructed into a three-dimensional face, and the target two-dimensional image can be obtained by offline shooting, cloud downloading, or calling from a local gallery.

[0047] S102, inputting the target two-dimensional image into a trained face reconstruction model, and outputting a three-dimensional face model corresponding to the target two-dimensional image.

[0048] Specifically, the target two-dimensional image is input into a trained face reconstruction model, and the face reconstruction reconstructs a three-dimensional face model corresponding to the target two-dimensional image based on the face image information in the target two-dimensional image.

[0049] Optionally, before inputting the target two-dimensional image into the trained face reconstruction model, the target two-dimensional image is preprocessed, and the preprocessing may include: if the target two-dimensional image does not have the face facing upward, rotating the target two-dimensional image so that the face in the target two-dimensional image is facing upward; cropping the target two-dimensional image to crop out the background area outside the face area to reduce the amount of data required to calculate the face reconstruction model; normalizing the cropped target two-dimensional image to normalize the grayscale value of each pixel in the target two-dimensional image to facilitate calculation of the face reconstruction model.

[0050] Optionally, the face reconstruction model may include a backbone based on a convolutional neural network (CNN) and a multi-layer perceptron (MLP) model. The backbone is a CNN-based encoder network for extracting image features from a target two-dimensional image. The backbone may be a backbone network such as, but not limited to, mobilenet, resnet, xception, etc. Then, the target two-dimensional image is input into a trained face reconstruction model, and a three-dimensional face model corresponding to the target two-dimensional image is output, including: inputting the target two-dimensional image into a CNN-based backbone network, extracting image features from the backbone target two-dimensional image, and then inputting the extracted image features into an MLP model, and the MLP model outputs regression parameters, transforms the regression parameters, generates a deformation parameter matrix, and then obtains a three-dimensional face model corresponding to the target two-dimensional image based on the deformation parameter matrix.

[0051] See also Figure 2 , is an example schematic diagram of a three-dimensional face reconstruction provided in an embodiment of the present application.

[0052] like Figure 2 As shown, the target two-dimensional image is input into the face reconstruction model to obtain the three-dimensional face model shown in the figure. The face reconstruction model shown includes a backbone based on a convolutional neural network (CNN) and a multi-layer perceptron (MLP) model.

[0053] It can be understood that the face reconstruction model is obtained based on complex training. In an embodiment of the present application, the face reconstruction model is generated based on cost function training, and the cost function includes a first cost function, a second cost function and a third cost function. The first cost function is obtained based on the predicted three-dimensional face model corresponding to the sample two-dimensional image and the standard three-dimensional face model corresponding to the sample two-dimensional image. The second cost function is obtained based on the predicted three-dimensional face model and the smoothed three-dimensional face model after smoothing the predicted three-dimensional face model. The third cost function is obtained based on the facial key points after two-dimensional projection of the predicted three-dimensional face model and the facial key points in the sample two-dimensional image. The predicted three-dimensional face model is obtained by predicting the sample two-dimensional image based on the created initial face reconstruction model, and the standard three-dimensional face model is obtained by constraining the reconstruction of the average face three-dimensional model based on the facial key points in the sample two-dimensional image.

[0054] The three-dimensional face model reconstruction method provided in the embodiment of the present application is adopted, the acquired target two-dimensional image is input into the face reconstruction model generated by training based on the cost function, and then the three-dimensional face model corresponding to the target two-dimensional image is output, the face reconstruction model is trained and generated using a first cost function constructed based on the predicted three-dimensional face model corresponding to the sample two-dimensional image and the standard three-dimensional face model corresponding to the sample two-dimensional image, a second cost function constructed based on the predicted three-dimensional face model and the smoothed three-dimensional face model after the predicted three-dimensional face model is smoothed, and a third cost function constructed based on the face key points after the predicted three-dimensional face model is two-dimensionally projected and the face key points in the sample two-dimensional image. Since the face reconstruction model is smoothed during the training process, the three-dimensional face model reconstructed using the face reconstruction model has an outstanding smoothing effect, and a good smoothing effect can be achieved without adding additional smoothing calculations during the application of the face reconstruction model.

[0055] See also Figure 3 , is a flow chart of a method for reconstructing a three-dimensional face model provided in an embodiment of the present application. Figure 3 As shown, the three-dimensional face model reconstruction method may include the following steps.

[0056] S201, collecting a two-dimensional image of a sample;

[0057] The sample two-dimensional images include human face images, and the sample two-dimensional images are used as training data to assist in training a three-dimensional human face model.

[0058] Optionally, the facial image area in the sample two-dimensional image should exceed a certain proportion of the total area of ​​the sample two-dimensional image, and the proportion may be 75%.

[0059] Optionally, if the facial image area in the sample two-dimensional image does not exceed the preset proportion of the total area of ​​the sample two-dimensional image, the sample two-dimensional image is sheared to obtain a new sample two-dimensional image, so that the facial image area in the new sample two-dimensional image exceeds the preset proportion of the total area of ​​the sample two-dimensional image.

[0060] Optionally, if the face image in the sample two-dimensional image is skewed, the sample two-dimensional image is rotated to obtain a new sample two-dimensional image, so that the face image in the new sample two-dimensional image is a frontal face image.

[0061] The sample two-dimensional image can be collected by offline shooting.

[0062] S202, extracting facial key points in the sample two-dimensional image and obtaining an average face three-dimensional model;

[0063] Specifically, facial key points are detected on the sample two-dimensional image to obtain facial key points in the sample two-dimensional image, and an average face three-dimensional model is obtained.

[0064] The key points of the face refer to some important feature points such as eyes, nose tip, mouth corners, eyebrows and contour points of various face parts.

[0065] The facial key point detection refers to locating the key area positions of the face and extracting the key points of the key area positions. The facial key point detection method includes but is not limited to the Active Shape Model (ASM) algorithm, the Active Appearance Model (AAM) algorithm and the deep learning algorithm.

[0066] The average face 3D model is obtained by using a depth camera to capture the face of a real person, obtaining the 3D point cloud data of the face, and then generating the face 3D model based on the 3D point cloud data. It is necessary to collect multiple real people and generate multiple face 3D models, and finally average the multiple face 3D models to obtain the average face 3D model.

[0067] S203, constraining the average face three-dimensional model based on the facial key points in the sample two-dimensional image to obtain a standard three-dimensional face model corresponding to the sample two-dimensional image;

[0068] Specifically, an initial deformation matrix and projection parameters are obtained, the average face three-dimensional model is reconstructed based on the deformation matrix to obtain an initial three-dimensional face model, the initial three-dimensional face model is projected into a two-dimensional space based on the projection parameters to obtain a two-dimensional face image, facial key points in the two-dimensional face image are extracted, and each facial key point in the two-dimensional face image is compared with each facial key point in the sample two-dimensional image to obtain a difference, the deformation matrix and projection parameters are updated based on the difference, the above process is iteratively performed until the difference between each facial key point in the two-dimensional face image and each facial key point in the sample two-dimensional image is less than a preset threshold, the iteration is stopped, and the deformation matrix at this time is the desired deformation matrix, and the average face three-dimensional model is constrainedly reconstructed based on the deformation matrix to obtain a standard three-dimensional face model corresponding to the sample two-dimensional image.

[0069] Optionally, after constraining the reconstruction of the average face 3D model based on the facial key points in the sample 2D image to obtain a standard 3D face model, the standard 3D face model is smoothed to obtain a new standard 3D face model, so that the newly obtained standard 3D face model has a better smoothing effect. In this way, in step S207, based on the first cost function constructed based on each vertex in the predicted 3D face model and each vertex in the standard 3D face model, the role played by the first cost function in the training process is not limited to the vertex correction of the face reconstruction model, but can also enhance the smoothing effect of the face reconstruction model.

[0070] Specifically, the step of smoothing the standard 3D face model to obtain a new standard 3D face model includes:

[0071]

[0072] Among them, the V i is the i-th 3D vertex in the standard 3D face model before smoothing, For the V i The directly adjacent first-order neighboring point, i represents the i-th three-dimensional vertex, j represents the number of the first-order neighboring point, is the i-th 3D vertex in the standard 3D face model after smoothing, and α represents the smoothing coefficient, α∈(0,1).

[0073] Optionally, constraining the reconstruction of the average face three-dimensional model based on the facial key points in the sample two-dimensional image to obtain a standard three-dimensional face model corresponding to the sample two-dimensional image can also be achieved by using an energy function to find the optimal value. Please refer to the following steps:

[0074] First, create the initial deformation matrix and projection parameters. Then iterate according to the following process.

[0075] Formula 1:min R,t,P′,w,Π E def (P′,w)+λE lan (Π,R,t,P′)

[0076] Among them, w is the deformation matrix, Π is the scale parameter in the projection parameters, R is the rotation parameter in the projection parameters, t is the translation parameter in the projection parameters, and P′ is the coordinate of the point in the standard three-dimensional face model. This formula can be used to fix the deformation matrix and projection parameters, and obtain the coordinate P′ of the point in the standard three-dimensional face model.

[0077] Determine the difference between this iteration and the previous iteration def Is the difference less than the set threshold, that is, |E def j -E def j-1 |, where j represents the number of iterations. If it is satisfied, the iteration is stopped. If it is not satisfied, the following process is continued and iterated.

[0078] Formula 2: E def =||P′-wP|| 2

[0079] Where P is the coordinate of the midpoint of the average face 3D model, P′ is obtained using Formula 1, P′ in the formula is fixed, and the deformation matrix w is solved using the least squares method.

[0080] Formula 3:

[0081] Among them, q i is the coordinate of each key point in the sample two-dimensional image, p′ i is the sample two-dimensional image q i Corresponding to the coordinates of the points in the standard 3D face model, P′ obtained by formula 1 is substituted into formula 3 to update the projection parameters Π,R,t.

[0082] It can be understood that Formula 2 is essentially an update of the deformation matrix in each iteration process, and Formula 3 is an update of the projection parameters in each iteration process.

[0083] Optionally, the smoothing of the standard 3D face model to obtain the standard 3D face model may also be based on curvature smoothing, Taubin smoothing algorithm, etc. The specific implementation method of the smoothing process is not limited in the embodiments of the present application.

[0084] S204, creating an initial face reconstruction model, and inputting the sample two-dimensional image into the initial face reconstruction model for prediction to obtain a predicted three-dimensional face model;

[0085] The initial face reconstruction model refers to an untrained, rough deep learning model.

[0086] Specifically, an initial face reconstruction model is created, a sample two-dimensional image is input into the initial face reconstruction model, and a predicted three-dimensional face model is obtained by using the initial face model for prediction.

[0087] It can be understood that since the initial face reconstruction model is an untrained, rough deep learning model, the predicted three-dimensional face model has a high degree of distortion and poor smoothness.

[0088] S205, performing smoothing processing on the predicted three-dimensional face model to obtain a smoothed three-dimensional face model;

[0089] For details, please refer to the specific implementation method of performing smoothing processing on the standard 3D face model to obtain a new standard 3D face model in step S203.

[0090] S206, performing two-dimensional projection on the predicted three-dimensional face model to obtain a predicted two-dimensional image, and extracting key points of the face in the predicted two-dimensional image;

[0091] Specifically, the projection parameters in the initial face reconstruction model are obtained, the predicted three-dimensional face model is two-dimensionally projected based on the projection parameters to obtain a predicted two-dimensional image, and the face key points in the predicted two-dimensional image are extracted.

[0092] It can be understood that the embodiment of the present application is essentially an iterative training process. During each iteration, the parameters of the initial face reconstruction model will be adjusted and updated, and the updated initial face reconstruction model will be used for the next round of iteration. After the parameters of the initial face reconstruction model are adjusted and updated, the updated initial face reconstruction model will also update the projection parameters.

[0093] S207, constructing a cost function based on the predicted three-dimensional face model, the standard three-dimensional face model, the smoothed three-dimensional face model, the facial key points in the predicted two-dimensional image, and the facial key points in the sample two-dimensional image;

[0094] Indicatively, in Figure 3 On the basis of Figure 4 As shown, step S207 may include S2071, S2072, S2073 and S2074.

[0095] S2071, constructing a first cost function based on each vertex in the predicted three-dimensional face model and each vertex in the standard three-dimensional face model;

[0096]

[0097] Wherein, N is the number of vertices, V i is the i-th vertex in the predicted three-dimensional face model, and the V GT,i is the i-th vertex in the standard three-dimensional face model.

[0098] It is not difficult to understand that the predicted three-dimensional face model is obtained by directly predicting the sample two-dimensional image by the initial face reconstruction model, and the standard three-dimensional face model can be obtained by constraining the reconstruction of the average face three-dimensional model based on the face key points in the sample two-dimensional image and then smoothing it. Therefore, compared with the predicted three-dimensional face model, the standard three-dimensional face model has higher accuracy in model shape and better smoothing effect. Furthermore, a first cost function is constructed based on each vertex in the predicted three-dimensional face model and each vertex in the standard three-dimensional face model, and the parameters of the initial face reconstruction model are adjusted and trained based on the first cost function, so as to improve the accuracy and smoothing effect of the reconstruction model by the initial face reconstruction model.

[0099] S2072, constructing a second cost function based on each vertex in the predicted three-dimensional face model and each vertex in the smoothed three-dimensional face model;

[0100]

[0101] Where N is the number of vertices, V i is the i-th vertex in the predicted three-dimensional face model, is the i-th vertex in the smoothed 3D face model.

[0102] It is not difficult to understand that the predicted three-dimensional face model is obtained by directly predicting the sample two-dimensional image by the initial face reconstruction model, and the smoothed three-dimensional face model is obtained by smoothing the predicted three-dimensional face model. Therefore, the smoothed three-dimensional face model has a significant smoothing effect compared to the predicted three-dimensional face model. Then, a second cost function is constructed based on each vertex in the predicted three-dimensional face model and each vertex in the smoothed three-dimensional face model, and the parameters of the initial face reconstruction model are adjusted and trained based on the second cost function, so as to optimize the smoothing effect of the initial face reconstruction model on the reconstruction model.

[0103] S2073, constructing a third cost function based on the facial key points in the predicted two-dimensional image and the facial key points in the sample two-dimensional image;

[0104]

[0105] Wherein, M is the number of key points of the face, and lmk i is the i-th face key point in the predicted two-dimensional image, and the lmk GT,i is the i-th facial key point in the sample two-dimensional image.

[0106] It is not difficult to understand that the predicted two-dimensional image is obtained by two-dimensionally projecting the predicted three-dimensional face model. Therefore, compared with the sample two-dimensional image, the predicted two-dimensional image has a large difference in the key points of the face. The third cost function is constructed based on the key points of the face in the predicted two-dimensional image and the key points of the face in the sample two-dimensional image. The adjustment and training of the parameters within the model of the initial face reconstruction model based on the third cost function can optimize the accuracy of the initial face reconstruction model to the reconstruction model, so that the reconstructed three-dimensional face model is closer to the standard three-dimensional face model corresponding to the sample two-dimensional image.

[0107] S2074: Perform weighted summation on the first cost function, the second cost function, and the third cost function to obtain a cost function.

[0108] Specifically, different weight values ​​are set for the first cost function, the second cost function and the third cost function, and weighted sum is performed to obtain a total cost function including the three cost functions.

[0109] Further, in the embodiment of the present application, a cost function based on MSE is used for calculation and back propagation. Optionally, other cost functions different from MSE can also be used, which is not limited in the embodiment of the present application. For example, a cost function based on SAD can be used, and the first cost function, the second cost function, and the third cost function can be respectively expressed as:

[0110] The first cost function:

[0111] The second cost function:

[0112] The third cost function:

[0113] S208, training the initial face reconstruction model based on the cost function to obtain a trained face reconstruction model;

[0114] It should be noted that, in the embodiment of the present application, steps S204 to S208 are an iterative loop training process, and steps S201 to S203 are used to provide training data for the iterative loop training process of steps S204 to S208.

[0115] See also Figure 5 , which is a flowchart of reconstructing a standard three-dimensional face model provided in an embodiment of the present application.

[0116] like Figure 5 As shown, by extracting key points of the face from the sample two-dimensional image, the three-dimensional model of the average face is constrained and reconstructed using the key points of the face extracted from the sample two-dimensional image and then smoothed to obtain a standard three-dimensional face model.

[0117] See also Figure 6 , which is a flowchart of training a face reconstruction model provided in an embodiment of the present application.

[0118] like Figure 6 As shown, the sample two-dimensional image is input into the initial face reconstruction model, and the initial face reconstruction model predicts a predicted three-dimensional face model. Figure 6 The standard three-dimensional face model obtained in the flowchart is used to construct the first cost function; the predicted three-dimensional face model is smoothed to obtain a smooth three-dimensional face model, and a second cost function is constructed based on the smooth three-dimensional face model and the predicted three-dimensional face model; the predicted three-dimensional face model is combined with the projection parameters given by the initial face reconstruction model to be projected into two dimensions to obtain a predicted two-dimensional image, and the face key points in the predicted two-dimensional image are extracted, as shown in the figure, face key point 2, and face key point 1 is a face key point in the sample two-dimensional image, and a third cost function is constructed based on face key point 2 in the predicted two-dimensional image and face key point 1 in the sample two-dimensional image; the first cost function, the second cost function and the third cost function are weighted summed and the initial face reconstruction model is updated. The above process is a complete training process, which is an iterative training process. The end mark of the training can be that the number of iterations reaches a preset number or the weighted sum of the first cost function, the second cost function and the third cost function is less than a preset threshold, and the face reconstruction model can be obtained after the training is completed.

[0119] S209, acquiring a target two-dimensional image;

[0120] S210, inputting the target two-dimensional image into a trained face reconstruction model, and outputting a three-dimensional face model corresponding to the target two-dimensional image.

[0121] In an embodiment of the present application, an initial face reconstruction model is first created, and a sample two-dimensional image is collected. Then, facial key points in the sample two-dimensional image are extracted, and an average face three-dimensional model is obtained. Based on the facial key points in the sample two-dimensional image, the average face three-dimensional model is constrained and reconstructed and smoothed to obtain a standard three-dimensional face model corresponding to the sample two-dimensional image. Then, the sample two-dimensional image is input into the initial face reconstruction model for prediction to obtain a predicted three-dimensional face model. The predicted three-dimensional face model is smoothed to obtain a smoothed three-dimensional face model. Then, the predicted three-dimensional face model is two-dimensionally projected to obtain a predicted two-dimensional image. Facial key points in the predicted two-dimensional image are further extracted. Finally, based on the The predicted three-dimensional face model, the standard three-dimensional face model, the smoothed three-dimensional face model, the face key points in the predicted two-dimensional image and the face key points in the sample two-dimensional image are used to construct three cost functions. The initial face reconstruction model is iteratively trained using the three cost functions to finally obtain a trained face reconstruction model. During the iterative training process, the smoothing effect and the accuracy of the model shape are judged, so that the trained face reconstruction model has good accuracy and outstanding smoothing effect for the reconstruction of the three-dimensional face model. Then, the trained face reconstruction model can be used to reconstruct a three-dimensional face model with high accuracy and significant smoothing effect for the target two-dimensional image, and the reconstructed three-dimensional face model is output.

[0122] By adopting the three-dimensional face model reconstruction method provided in the embodiment of the present application, the face reconstruction model is trained based on smoothing effect and accuracy. The trained face reconstruction model can be used to predict the target two-dimensional image and reconstruct a three-dimensional face model with significant smoothing effect and accuracy. The face reconstruction model only needs to be trained once and can be reused for any target two-dimensional image. During use, the three-dimensional face model can be obtained by simply inputting the target two-dimensional image into the face reconstruction model. The target two-dimensional image and the three-dimensional face model are generated end-to-end without adding additional computational effort.

[0123] See also Figure 7 , is a flow chart of a method for reconstructing a three-dimensional face model provided in an embodiment of the present application. Figure 7 As shown, the three-dimensional face model reconstruction method includes the following steps.

[0124] S301, acquiring a target two-dimensional image;

[0125] S302, inputting the target two-dimensional image into a trained face reconstruction model, and outputting a three-dimensional face model corresponding to the target two-dimensional image;

[0126] Wherein, the face reconstruction model is generated based on cost function training, and the cost function includes a first cost function, a second cost function and a third cost function, the first cost function is obtained based on a predicted three-dimensional face model corresponding to a sample two-dimensional image and a standard three-dimensional face model corresponding to the sample two-dimensional image, the second cost function is obtained based on the predicted three-dimensional face model and a smoothed three-dimensional face model after smoothing the predicted three-dimensional face model, and the third cost function is obtained based on facial key points after two-dimensional projection of the predicted three-dimensional face model and facial key points in the sample two-dimensional image;

[0127] The predicted three-dimensional face model is obtained by predicting the sample two-dimensional image based on the created initial face reconstruction model, and the standard three-dimensional face model is obtained by constraining and reconstructing the average face three-dimensional model based on the face key points in the sample two-dimensional image.

[0128] S303: Smoothing the three-dimensional face model to obtain a target three-dimensional face model.

[0129] Specifically, they include:

[0130]

[0131] Among them, the V i ′ is the i-th three-dimensional vertex in the three-dimensional face model before smoothing. For the V i ' is the first-order neighboring point directly adjacent to the vertex, i represents the i-th three-dimensional vertex, j represents the number of the first-order neighboring point, is the i-th 3D vertex in the target 3D face model after smoothing, and α represents a smoothing coefficient, α∈(0,1).

[0132] The three-dimensional face model reconstruction method provided in the embodiment of the present application is adopted, the acquired target two-dimensional image is input into the face reconstruction model generated by training based on the cost function, and then the three-dimensional face model corresponding to the target two-dimensional image is output, and the face reconstruction model is trained and generated using a first cost function constructed based on the predicted three-dimensional face model corresponding to the sample two-dimensional image and the standard three-dimensional face model corresponding to the sample two-dimensional image, a second cost function constructed based on the predicted three-dimensional face model and the smoothed three-dimensional face model after the predicted three-dimensional face model is smoothed, and a third cost function constructed based on the face key points after the predicted three-dimensional face model is two-dimensionally projected and the face key points in the sample two-dimensional image. Since the face reconstruction model is smoothed during the training process, the three-dimensional face model reconstructed using the face reconstruction model has an outstanding smoothing effect, and in the application process of the face reconstruction model, it is not necessary to increase the amount of additional smoothing calculations to output the three-dimensional face model corresponding to the target two-dimensional image with a good smoothing effect, and after the face reconstruction model outputs the three-dimensional face model, the output three-dimensional face model is smoothed to obtain the target three-dimensional face model, which further increases the smoothing effect.

[0133] Optionally, stylization is added during the training process of the face reconstruction model so that the face reconstruction model can reconstruct a stylized three-dimensional face model.

[0134] See also Figure 8 , is a flow chart of a method for reconstructing a three-dimensional face model provided in an embodiment of the present application. Figure 8 As shown, the three-dimensional face model reconstruction method includes the following steps.

[0135] S401, collecting a two-dimensional image of a sample;

[0136] S402, performing stylization processing on the sample two-dimensional image to obtain a stylized sample two-dimensional image, and extracting facial key points in the stylized sample two-dimensional image;

[0137] The stylization processing refers to converting the sample two-dimensional image into a specific stylized sample two-dimensional image, such as a sketch portrait style, a cartoon image (animation) style, an oil painting style, etc.

[0138] See also Fig. 9 , which is an example schematic diagram of a stylized processing provided in an embodiment of the present application.

[0139] like Fig. 9 As shown, the sample two-dimensional image shown can be processed by cartoon image stylization to obtain a stylized sample two-dimensional image of a cartoon image as shown in the figure.

[0140] S403, obtaining an average face three-dimensional model;

[0141] S404, constraining the average face three-dimensional model based on the facial key points in the stylized sample two-dimensional image to obtain a standard three-dimensional face model corresponding to the sample two-dimensional image;

[0142] See also Fig.10 , is an example schematic diagram of reconstructing a three-dimensional face model using facial key point constraints provided in an embodiment of the present application.

[0143] like Fig.10 As shown, in the process of reconstructing the three-dimensional face model, the average face three-dimensional model is constrained by the facial key points in the stylized sample two-dimensional image to obtain a standard three-dimensional face model corresponding to the stylized sample two-dimensional image.

[0144] S405, creating an initial face reconstruction model, and inputting the sample two-dimensional image into the initial face reconstruction model for prediction to obtain a predicted three-dimensional face model;

[0145] S406, performing smoothing processing on the predicted three-dimensional face model to obtain a smoothed three-dimensional face model;

[0146] S407, performing two-dimensional projection on the predicted three-dimensional face model to obtain a predicted two-dimensional image, and extracting key points of the face in the predicted two-dimensional image;

[0147] S408, constructing a cost function based on the predicted three-dimensional face model, the standard three-dimensional face model, the smoothed three-dimensional face model, the facial key points in the predicted two-dimensional image, and the facial key points in the stylized sample two-dimensional image;

[0148] S409, training the initial face reconstruction model based on the cost function to obtain a trained face reconstruction model;

[0149] S410, acquiring a target two-dimensional image;

[0150] S411, inputting the target two-dimensional image into a trained face reconstruction model, and outputting a stylized three-dimensional face model corresponding to the target two-dimensional image.

[0151] By adopting the three-dimensional face model reconstruction method provided in the embodiment of the present application, stylization processing is added in the process of training the face reconstruction model. The face reconstruction model trained based on smoothing effect and accuracy can output a stylized three-dimensional face model, thereby increasing the interest and functionality of the three-dimensional face model reconstruction.

[0152] See also Fig.11 , is a schematic diagram of the structure of a three-dimensional face model reconstruction device provided in an embodiment of the present application. Fig.11As shown, the 3D face model reconstruction device 1 can be implemented as all or part of a computer device through software, hardware or a combination of both. According to some embodiments, the 3D face model reconstruction device 1 includes an image acquisition module 11 and a model prediction module 12, specifically including:

[0153] An image acquisition module 11 is used to acquire a target two-dimensional image;

[0154] A model prediction module 12 is used to input the target two-dimensional image into a trained face reconstruction model and output a three-dimensional face model corresponding to the target two-dimensional image;

[0155] Wherein, the face reconstruction model is generated based on cost function training, and the cost function includes a first cost function, a second cost function and a third cost function, the first cost function is obtained based on a predicted three-dimensional face model corresponding to a sample two-dimensional image and a standard three-dimensional face model corresponding to the sample two-dimensional image, the second cost function is obtained based on the predicted three-dimensional face model and a smoothed three-dimensional face model after smoothing the predicted three-dimensional face model, and the third cost function is obtained based on facial key points after two-dimensional projection of the predicted three-dimensional face model and facial key points in the sample two-dimensional image;

[0156] The predicted three-dimensional face model is obtained by predicting the sample two-dimensional image based on the created initial face reconstruction model, and the standard three-dimensional face model is obtained by constraining and reconstructing the average face three-dimensional model based on the face key points in the sample two-dimensional image.

[0157] Optionally, the model prediction module 12 is specifically used for:

[0158] Inputting the target two-dimensional image into a trained face reconstruction model, and outputting a stylized three-dimensional face model corresponding to the target two-dimensional image;

[0159] The face reconstruction model is generated based on cost function training, and the cost function includes a first cost function, a second cost function and a third cost function. The first cost function is obtained based on a predicted stylized three-dimensional face model corresponding to a sample two-dimensional image and a standard stylized three-dimensional face model corresponding to the sample two-dimensional image. The second cost function is obtained based on the predicted stylized three-dimensional face model and a smoothed stylized three-dimensional face model after smoothing the predicted stylized three-dimensional face model. The third cost function is obtained based on facial key points after two-dimensional projection of the predicted stylized three-dimensional face model and facial key points in a stylized sample two-dimensional image corresponding to the sample two-dimensional image.

[0160] The predicted stylized 3D face model is obtained by predicting the sample 2D image based on the created initial face reconstruction model, and the standard stylized 3D face model is obtained by constraining the reconstruction of the average face 3D model based on the face key points in the stylized sample 2D image corresponding to the sample 2D image.

[0161] Optional, see Fig.12 , is a schematic diagram of the structure of a three-dimensional face model reconstruction device provided in an embodiment of the present application. Fig.12 As shown, the device also includes:

[0162] The smoothing processing module 13 is used to perform smoothing processing on the three-dimensional face model to obtain a target three-dimensional face model.

[0163] Optionally, the smoothing processing module 13 is specifically used for:

[0164]

[0165] Among them, the V i ′ is the i-th three-dimensional vertex in the three-dimensional face model before smoothing. For the V i ' is the first-order neighboring point directly adjacent to the vertex, i represents the i-th three-dimensional vertex, j represents the number of the first-order neighboring point, is the i-th 3D vertex in the target 3D face model after smoothing, and α represents a smoothing coefficient, α∈(0,1).

[0166] Optional, such as Fig.12 As shown, the device also includes a model training module 14.

[0167] Optional, see Fig.13 , is a schematic diagram of the structure of a model training module provided in an embodiment of the present application. Fig.13 As shown, the model training module 14 includes:

[0168] The image acquisition unit 141 is used to create an initial face reconstruction model and acquire a sample two-dimensional image;

[0169] A first key point extraction unit 142, used to extract key points of the face in the sample two-dimensional image and obtain an average face three-dimensional model;

[0170] A model reconstruction unit 143, configured to perform constrained reconstruction on the average face three-dimensional model based on the facial key points in the sample two-dimensional image to obtain a standard three-dimensional face model corresponding to the sample two-dimensional image;

[0171] A model prediction unit 144, configured to input the sample two-dimensional image into the initial face reconstruction model for prediction to obtain a predicted three-dimensional face model;

[0172] A smoothing processing unit 145, configured to perform smoothing processing on the predicted 3D face model to obtain a smoothed 3D face model;

[0173] A second key point extraction unit 146, configured to perform two-dimensional projection on the predicted three-dimensional face model to obtain a predicted two-dimensional image, and extract key points of the face in the predicted two-dimensional image;

[0174] A cost function construction unit 147, configured to construct a cost function based on the predicted 3D face model, the standard 3D face model, the smoothed 3D face model, the face key points in the predicted 2D image, and the face key points in the sample 2D image;

[0175] The model training unit 148 is used to train the initial face reconstruction model based on the cost function to obtain a trained face reconstruction model.

[0176] Optionally, the first key point extraction unit 142 is specifically configured to:

[0177] Performing stylization processing on the sample two-dimensional image to obtain a stylized sample two-dimensional image, and extracting key points of a human face in the stylized sample two-dimensional image;

[0178] The cost function construction unit 147 is specifically used for:

[0179] A cost function is constructed based on the predicted three-dimensional face model, the standard three-dimensional face model, the smoothed three-dimensional face model, the facial key points in the predicted two-dimensional image, and the facial key points in the stylized sample two-dimensional image.

[0180] Optionally, the second key point extraction unit 146 is specifically configured to:

[0181] The projection parameters in the initial face reconstruction model are obtained, the predicted three-dimensional face model is two-dimensionally projected based on the projection parameters to obtain a predicted two-dimensional image, and the face key points in the predicted two-dimensional image are extracted.

[0182] Optional, see Fig.14 , is a structural diagram of a cost function construction unit provided in an embodiment of the present application.

[0183] like Fig.14 As shown, the cost function construction unit 147 includes:

[0184] The first construction subunit 1471 is used to construct a first cost function based on each vertex in the predicted three-dimensional face model and each vertex in the standard three-dimensional face model: Wherein, N is the number of vertices, V i is the i-th vertex in the predicted three-dimensional face model, and the V GT,i is the i-th vertex in the standard three-dimensional face model;

[0185] The second construction subunit 1472 is used to construct a second cost function based on each vertex in the predicted 3D face model and each vertex in the smoothed 3D face model: Where N is the number of vertices, V i is the i-th vertex in the predicted three-dimensional face model, is the i-th vertex in the smoothed three-dimensional face model;

[0186] The third construction subunit 1473 is used to construct a third cost function based on the facial key points in the predicted two-dimensional image and the facial key points in the sample two-dimensional image: Wherein M is the number of key points of the face, and lmk i is the i-th face key point in the predicted two-dimensional image, and the lmk GT,i is the i-th facial key point in the sample two-dimensional image;

[0187] The weighted summation subunit 1474 is configured to perform weighted summation on the first cost function, the second cost function and the third cost function to obtain a cost function.

[0188] Optionally, the weighted summation subunit 1474 is specifically configured to:

[0189] L=β·L1+μ·L2+λ·L3

[0190] Wherein, the β is the weight corresponding to the first cost function L1, the μ is the weight corresponding to the second cost function L2, and the λ is the weight corresponding to the third cost function L3.

[0191] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0192] Using the three-dimensional face model reconstruction method provided in an embodiment of the present application, the acquired target two-dimensional image is input into a face reconstruction model generated based on cost function training, and then a three-dimensional face model corresponding to the target two-dimensional image is output. The face reconstruction model is trained and generated using a first cost function constructed based on a predicted three-dimensional face model corresponding to a sample two-dimensional image and a standard three-dimensional face model corresponding to the sample two-dimensional image, a second cost function constructed based on the predicted three-dimensional face model and a smoothed three-dimensional face model after smoothing the predicted three-dimensional face model, and a third cost function constructed based on facial key points after two-dimensional projection of the predicted three-dimensional face model and facial key points in the sample two-dimensional image. Since the face reconstruction model is smoothed during the training process, the three-dimensional face model reconstructed using the face reconstruction model has an outstanding smoothing effect, and in the application process of the face reconstruction model, no additional smoothing calculation amount is required to output a three-dimensional face model with a good smoothing effect corresponding to the target two-dimensional image, and after the face reconstruction model outputs the three-dimensional face model, the output three-dimensional face model is smoothed to obtain the target three-dimensional face model, which further increases the smoothing effect; optionally, stylization processing can be added during the training of the face reconstruction model, and the face reconstruction model trained based on the smoothing effect and accuracy can output a stylized three-dimensional face model, which increases the fun and functionality of the three-dimensional face model reconstruction.

[0193] The present application also provides a computer storage medium, which can store multiple instructions, and the instructions are suitable for being loaded and executed by a processor as described above. Figures 1 to 10 The three-dimensional face model reconstruction method of the embodiment shown in the figure can be specifically implemented by referring to Figures 1 to 10 The specific description of the illustrated embodiment will not be repeated here.

[0194] The present application also provides a computer program product, which stores at least one instruction, and the at least one instruction is loaded and executed by the processor as described above. Figures 1 to 10 The three-dimensional face model reconstruction method of the embodiment shown in the figure can be specifically implemented by referring to Figures 1 to 10 The specific description of the illustrated embodiment will not be repeated here.

[0195] Please refer to Fig.15 , which shows a block diagram of a computer device provided by an exemplary embodiment of the present application. The computer device in the present application may include one or more of the following components: a processor 110, a memory 120, an input device 130, an output device 140, and a bus 150. The processor 110, the memory 120, the input device 130, and the output device 140 may be connected via a bus 150.

[0196] The processor 110 may include one or more processing cores. The processor 110 uses various interfaces and lines to connect various parts of the entire computer device, and executes various functions and processes data of the computer device 100 by running or executing instructions, programs, code sets or instruction sets stored in the memory 120, and calling data stored in the memory 120. Optionally, the processor 110 can be implemented in at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), and programmable logic array (PLA). The processor 110 can integrate one or a combination of a central processing unit (CPU), a graphics processing unit (GPU), and a modem. Among them, the CPU mainly processes the operating system, user interface, and application programs; the GPU is responsible for rendering and drawing display content; and the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 110, but may be implemented separately through a communication chip.

[0197] The memory 120 may include a random access memory (RAM) or a read-only memory (ROM). Optionally, the memory 120 includes a non-transitory computer-readable storage medium. The memory 120 may be used to store instructions, programs, codes, code sets, or instruction sets.

[0198] The input device 130 is used to receive input instructions or data, and the input device 130 includes but is not limited to a keyboard, a mouse, a camera, a microphone or a touch device. The output device 140 is used to output instructions or data, and the output device 140 includes but is not limited to a display device and a speaker. In the embodiment of the present application, the input device 130 can be a temperature sensor for obtaining the operating temperature of the computer device. The output device 140 can be a speaker for outputting audio signals.

[0199] In addition, those skilled in the art will appreciate that the structure of the computer device shown in the above drawings does not constitute a limitation on the computer device, and the computer device may include more or fewer components than shown, or combine certain components, or arrange the components differently. For example, the computer device also includes a radio frequency circuit, an input unit, a sensor, an audio circuit, a wireless fidelity (WiFi) module, a power supply, a Bluetooth module, and other components, which will not be described in detail here.

[0200] In the embodiment of the present application, the execution subject of each step may be the computer device described above. Optionally, the execution subject of each step is the operating system of the computer device. The operating system may be an Android system, an IOS system, or other operating systems, which is not limited in the embodiment of the present application.

[0201] exist Fig.15 In the computer device shown, the processor 110 can be used to call the three-dimensional face model reconstruction program stored in the memory 120 and execute it to implement the three-dimensional face model reconstruction method as described in the various method embodiments of the present application.

[0202] Using the three-dimensional face model reconstruction method provided in an embodiment of the present application, the acquired target two-dimensional image is input into a face reconstruction model generated based on cost function training, and then a three-dimensional face model corresponding to the target two-dimensional image is output. The face reconstruction model is trained and generated using a first cost function constructed based on a predicted three-dimensional face model corresponding to a sample two-dimensional image and a standard three-dimensional face model corresponding to the sample two-dimensional image, a second cost function constructed based on the predicted three-dimensional face model and a smoothed three-dimensional face model after smoothing the predicted three-dimensional face model, and a third cost function constructed based on facial key points after two-dimensional projection of the predicted three-dimensional face model and facial key points in the sample two-dimensional image. Since the face reconstruction model is smoothed during the training process, the three-dimensional face model reconstructed using the face reconstruction model has an outstanding smoothing effect, and in the application process of the face reconstruction model, no additional smoothing calculation amount is required to output a three-dimensional face model with a good smoothing effect corresponding to the target two-dimensional image, and after the face reconstruction model outputs the three-dimensional face model, the output three-dimensional face model is smoothed to obtain the target three-dimensional face model, which further increases the smoothing effect; optionally, stylization processing can be added during the training of the face reconstruction model, and the face reconstruction model trained based on the smoothing effect and accuracy can output a stylized three-dimensional face model, which increases the fun and functionality of the three-dimensional face model reconstruction.

[0203] Those skilled in the art can clearly understand that the technical solution of the present application can be implemented with the help of software and / or hardware. The "unit" and "module" in this specification refer to software and / or hardware that can independently complete or cooperate with other components to complete specific functions, where the hardware can be, for example, a field programmable gate array (FPGA), an integrated circuit (IC), etc.

[0204] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0205] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0206] In the several embodiments provided in the present application, it should be understood that the disclosed devices can be implemented in other ways. For example, the device embodiments described above are only schematic, such as the division of the units, which is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some service interfaces, and the indirect coupling or communication connection of devices or units can be electrical or other forms.

[0207] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0208] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0209] A person skilled in the art may understand that all or part of the steps in the various methods of the above embodiments may be completed by entering a program to instruct the relevant hardware. The program may be stored in a computer-readable memory, and the memory may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0210] The above is only an exemplary embodiment of the present disclosure, and the scope of the present disclosure cannot be limited thereto. That is, any equivalent changes and modifications made according to the teachings of the present disclosure are still within the scope of the present disclosure. After considering the specification and practicing the disclosure here, those skilled in the art will easily think of other embodiments of the present disclosure. This application is intended to cover any modification, use or adaptation of the present disclosure, which follows the general principles of the present disclosure and includes common knowledge or customary technical means in the technical field not recorded in the present disclosure. The description and examples are only regarded as exemplary, and the scope and spirit of the present disclosure are defined by the claims.

Claims

1. A method for reconstructing a three-dimensional face model, characterized in that: The method comprises: Collecting a two-dimensional image of the sample; Extracting key points of the human face in the sample two-dimensional image and obtaining an average face three-dimensional model; Based on the facial key points in the sample two-dimensional image, the average face three-dimensional model is constrainedly reconstructed to obtain a standard three-dimensional face model corresponding to the sample two-dimensional image; Creating an initial face reconstruction model, and inputting the sample two-dimensional image into the initial face reconstruction model for prediction to obtain a predicted three-dimensional face model; Smoothing the predicted three-dimensional face model to obtain a smoothed three-dimensional face model; Performing two-dimensional projection on the predicted three-dimensional face model to obtain a predicted two-dimensional image, and extracting key points of the face in the predicted two-dimensional image; Constructing a first cost function based on each vertex in the predicted three-dimensional face model and each vertex in the standard three-dimensional face model: Wherein, N is the number of vertices, V i is the i-th vertex in the predicted three-dimensional face model, and the V GT,i is the i-th vertex in the standard three-dimensional face model; Constructing a second cost function based on each vertex in the predicted three-dimensional face model and each vertex in the smoothed three-dimensional face model: Where N is the number of vertices, V i is the i-th vertex in the predicted three-dimensional face model, is the i-th vertex in the smoothed three-dimensional face model; Constructing a third cost function based on the facial key points in the predicted two-dimensional image and the facial key points in the sample two-dimensional image: Wherein M is the number of key points of the face, and lmk i is the i-th face key point in the predicted two-dimensional image, and the lmk GT,i is the i-th facial key point in the sample two-dimensional image; Performing weighted summation on the first cost function, the second cost function and the third cost function to obtain a cost function: L=β·L1+μ·L2+λ·L3, wherein β is the weight corresponding to the first cost function L1, μ is the weight corresponding to the second cost function L2, and λ is the weight corresponding to the third cost function L3; Training the initial face reconstruction model based on the cost function to obtain a trained face reconstruction model; Acquire a target two-dimensional image; Inputting the target two-dimensional image into the face reconstruction model, and outputting a three-dimensional face model corresponding to the target two-dimensional image; Execute the preset smoothing function: Get the target three-dimensional face model, where the V i ′ is the i-th three-dimensional vertex in the three-dimensional face model before smoothing, For the V i ′ The directly adjacent first-order neighboring points, i represents the i-th three-dimensional vertex, N i Indicates the The number of j represents the number of first-order neighboring points, the is the i-th 3D vertex in the target 3D face model after smoothing, and α represents a smoothing coefficient, α∈(0,1).

2. The method according to claim 1, characterized in that The step of inputting the target two-dimensional image into a trained face reconstruction model and outputting a three-dimensional face model corresponding to the target two-dimensional image comprises: Inputting the target two-dimensional image into a trained face reconstruction model, and outputting a stylized three-dimensional face model corresponding to the target two-dimensional image; The face reconstruction model is generated based on cost function training, and the cost function includes a first cost function, a second cost function and a third cost function. The first cost function is obtained based on a predicted stylized three-dimensional face model corresponding to a sample two-dimensional image and a standard stylized three-dimensional face model corresponding to the sample two-dimensional image. The second cost function is obtained based on the predicted stylized three-dimensional face model and a smoothed stylized three-dimensional face model after smoothing the predicted stylized three-dimensional face model. The third cost function is obtained based on facial key points after two-dimensional projection of the predicted stylized three-dimensional face model and facial key points in a stylized sample two-dimensional image corresponding to the sample two-dimensional image. The predicted stylized 3D face model is obtained by predicting the sample 2D image based on the created initial face reconstruction model, and the standard stylized 3D face model is obtained by constraining the reconstruction of the average face 3D model based on the face key points in the stylized sample 2D image corresponding to the sample 2D image.

3. The method according to claim 1, characterized in that The extracting of key points of the face in the sample two-dimensional image comprises: Performing stylization processing on the sample two-dimensional image to obtain a stylized sample two-dimensional image, and extracting key points of a human face in the stylized sample two-dimensional image; The cost function is constructed based on the predicted three-dimensional face model, the standard three-dimensional face model, the smoothed three-dimensional face model, the face key points in the predicted two-dimensional image, and the face key points in the sample two-dimensional image, including: A cost function is constructed based on the predicted three-dimensional face model, the standard three-dimensional face model, the smoothed three-dimensional face model, the facial key points in the predicted two-dimensional image, and the facial key points in the stylized sample two-dimensional image.

4. The method according to claim 1, characterized in that The step of performing two-dimensional projection on the predicted three-dimensional face model to obtain a predicted two-dimensional image, and extracting key points of the face in the predicted two-dimensional image, comprises: The projection parameters in the initial face reconstruction model are obtained, the predicted three-dimensional face model is two-dimensionally projected based on the projection parameters to obtain a predicted two-dimensional image, and the face key points in the predicted two-dimensional image are extracted.

5. A 3D face model reconstruction device, characterized in that: The device comprises: The model training module is used to collect sample two-dimensional images; extract facial key points in the sample two-dimensional images and obtain an average face three-dimensional model; constrain the average face three-dimensional model based on the facial key points in the sample two-dimensional image to obtain a standard three-dimensional face model corresponding to the sample two-dimensional image; create an initial face reconstruction model, and input the sample two-dimensional image into the initial face reconstruction model for prediction to obtain a predicted three-dimensional face model; smooth the predicted three-dimensional face model to obtain a smoothed three-dimensional face model; perform two-dimensional projection on the predicted three-dimensional face model to obtain a predicted two-dimensional image, and extract facial key points in the predicted two-dimensional image; construct a first cost function based on each vertex in the predicted three-dimensional face model and each vertex in the standard three-dimensional face model: Wherein, N is the number of vertices, V i is the i-th vertex in the predicted three-dimensional face model, and the V GT,i is the i-th vertex in the standard three-dimensional face model; constructing a second cost function based on each vertex in the predicted three-dimensional face model and each vertex in the smoothed three-dimensional face model: Where N is the number of vertices, V i is the i-th vertex in the predicted three-dimensional face model, is the i-th vertex in the smoothed three-dimensional face model; constructing a third cost function based on the face key points in the predicted two-dimensional image and the face key points in the sample two-dimensional image: Wherein M is the number of key points of the face, and lmk i is the i-th face key point in the predicted two-dimensional image, and the lmk GT,i is the i-th facial key point in the sample two-dimensional image; performing weighted summation on the first cost function, the second cost function and the third cost function to obtain a cost function: L=β·L1+μ·L2+λ·L3, wherein β is the weight corresponding to the first cost function L1, μ is the weight corresponding to the second cost function L2, and λ is the weight corresponding to the third cost function L3; An image acquisition module, used for acquiring a target two-dimensional image; A model prediction module, used to input the target two-dimensional image into a trained face reconstruction model, and output a three-dimensional face model corresponding to the target two-dimensional image; Smoothing module, used to execute preset smoothing functions: Get the target three-dimensional face model, where the V i ′ is the i-th three-dimensional vertex in the three-dimensional face model before smoothing, For the V i ′ The directly adjacent first-order neighboring points, i represents the i-th three-dimensional vertex, N i Indicates the The number of j represents the number of first-order neighboring points, the is the i-th 3D vertex in the target 3D face model after smoothing, and α represents a smoothing coefficient, α∈(0,1).

6. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.

7. A computer device, characterized in that: include: A processor and a memory; wherein the memory stores a computer program, and the computer program is suitable for being loaded by the processor and executing the steps of the method as claimed in any one of claims 1 to 4.

Images