Training method of parameter prediction model and three-dimensional face reconstruction method and device

Abstract

The application provides a parameter prediction model training method and a three-dimensional face reconstruction method and device, comprising: obtaining a two-dimensional face image training set and inputting the two-dimensional face image training set into a pre-constructed neural network model; extracting face features from the two-dimensional face image training set through a main network, and predicting a plurality of first face feature parameters based on the face features by using a plurality of detection branch networks; determining at least one face feature parameter to be refined from the plurality of first face feature parameters, and inputting the face feature parameter to be refined into a corresponding branch refining network to predict a second face feature parameter corresponding to the face feature parameter to be refined; and performing joint training on the neural network model according to a first loss corresponding to the first face feature parameter and a second loss corresponding to the second face feature parameter to obtain a trained neural network model. The application can improve the accuracy while ensuring that the computational complexity is not increased.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of machine learning, and in particular to a parameter prediction model training method and a three-dimensional face reconstruction method and device. BACKGROUND

[0002] 3DMM (3D Morphable Face Model) is a statistical model of face shape and appearance, which was first proposed to solve the problem of recovering three-dimensional shape from two-dimensional face images. At present, due to the wide use of neural networks, the 3DMM parameter optimization is simplified, that is, the optimized face feature parameters can be obtained through a neural network, and the three-dimensional face reconstruction is completed based on the optimized face feature parameters.

[0003] However, the existing neural network is based on a statistical model of a set of face shapes and textures to represent any face, and there is still a problem of difficult parameter solving, which makes the subsequent three-dimensional face reconstruction have the problem of poor discriminability of the reconstructed face.

[0004] In order to improve the discriminability of the reconstructed face, the accuracy of face feature parameter prediction needs to be improved, and how to train a neural network model to improve the prediction accuracy of face feature parameters without increasing the calculation amount of the neural network model is a problem to be solved. SUMMARY

[0005] The present application provides a parameter prediction model training method and a three-dimensional face reconstruction method and device to solve the above problems.

[0006] The present application provides a parameter prediction model training method, comprising:

[0007] Obtain a two-dimensional face image training set and input it into a pre-constructed neural network model; wherein the pre-constructed neural network model comprises a parameter prediction model and at least one branch refining network; the parameter prediction model comprises a backbone network and a plurality of detection branch networks;

[0008] Extract face features from the two-dimensional face image training set through the backbone network, and predict a plurality of first face feature parameters based on the face features through the plurality of detection branch networks;

[0009] Determine at least one face feature parameter to be refined from the plurality of first face feature parameters, and input the face feature parameter to be refined into the corresponding branch refining network to predict a second face feature parameter corresponding to the face feature parameter to be refined;

[0010] The neural network model is jointly trained based on the first loss corresponding to the first face feature parameters and the second loss corresponding to the second face feature parameters to obtain a trained neural network model. The trained parameter prediction model includes a trained parameter prediction model and a trained branch extraction network.

[0011] The first loss is obtained based on the ground truth value corresponding to the training set of the two-dimensional face images and the first face feature parameter; the second loss includes the loss between the ground truth value corresponding to the training set of the two-dimensional face images and the second face feature parameter, as well as the loss between the face feature parameter to be extracted and the second face feature parameter.

[0012] According to a training method for a parameter prediction model provided by the present invention, after extracting facial features from the two-dimensional face image training set through the backbone network and predicting multiple first facial feature parameters based on the facial features using the multiple detection branch networks, the method further includes:

[0013] Based on the first facial feature parameters, the reference facial coordinates, and the reference facial parameters, the first facial feature parameters are converted into facial key points;

[0014] Accordingly, the branching network includes m sequentially connected convolutional layers, and there is a skip connection between the m-th convolutional layer and one of the previous m-1 convolutional layers;

[0015] The step of inputting the facial feature parameters to be extracted into the corresponding branch extraction network to predict the second facial feature parameters corresponding to the facial feature parameters to be extracted includes:

[0016] The facial feature parameters to be refined are extracted through the first m-1 convolutional layers to obtain the feature information output by the m-1th convolutional layer;

[0017] The feature information output by the (m-1)th convolutional layer, the feature information output by the convolutional layer connected to the mth convolutional layer via skip connections, and the facial key points are input into the mth convolutional layer to output the second facial feature parameters.

[0018] According to a training method for a parameter prediction model provided by the present invention, the second loss includes the loss between the ground truth corresponding to the two-dimensional face image training set and the second face feature parameters, and the loss between the face feature parameters to be extracted and the second face feature parameters, including:

[0019] The loss between the ground truth and the second face feature parameters corresponding to the training set of the two-dimensional face image and the loss between the face feature parameters to be extracted and the second face feature parameters are calculated using the mean square error calculation method.

[0020] According to the training method of the parameter prediction model provided by the present invention, the first facial feature parameters include at least pose parameters, shape parameters and expression parameters;

[0021] The reference facial parameters include reference expression parameters and reference shape parameters;

[0022] Accordingly, the step of converting the first facial feature parameters into facial key points based on the first facial feature parameters, the reference facial coordinates, and the reference facial parameters includes:

[0023] Obtain the product between the expression parameter and the baseline expression parameter, and the product between the shape parameter and the baseline shape parameter, respectively;

[0024] Based on the baseline face coordinates, the sum of the product between the expression parameters and the baseline expression parameters and the product between the shape parameters and the baseline shape parameters is obtained as the deformation parameters of the expression and shape.

[0025] Obtain the product between the deformation parameters and pose parameters of the expression and shape, and use it as the facial keypoint under the first facial feature parameters.

[0026] According to the training method of the parameter prediction model provided by the present invention, the number of branch extraction networks is the same as the number of facial feature parameters to be extracted, and the branch extraction networks correspond one-to-one with the facial feature parameters to be extracted.

[0027] Accordingly, determining at least one facial feature parameter to be extracted from the plurality of first facial feature parameters includes:

[0028] The facial feature parameters to be extracted are determined from the first facial feature parameters according to the preset accuracy improvement target;

[0029] The facial feature parameters to be extracted are input into the corresponding branch extraction network.

[0030] This invention also provides a three-dimensional face reconstruction method, comprising:

[0031] Obtain the 2D face image to be reconstructed;

[0032] The two-dimensional face image to be reconstructed is input into a pre-trained parameter prediction model to predict multiple face feature parameters.

[0033] The multiple facial feature parameters are input into a three-dimensional deformation model to obtain the reconstructed three-dimensional face;

[0034] The pre-trained parameter prediction model is trained based on the above-mentioned parameter prediction model training method.

[0035] The present invention also provides a training apparatus for a parameter prediction model, comprising:

[0036] The training set acquisition and input module is used to acquire a two-dimensional face image training set and input it into a pre-built neural network model; wherein, the pre-built neural network model includes a parameter prediction model and at least one branch extraction network; the parameter prediction model includes a backbone network and multiple detection branch networks;

[0037] The parameter prediction model prediction module is used to extract facial features from the two-dimensional face image training set through the backbone network, and use the multiple detection branch networks to predict multiple first facial feature parameters based on the facial features.

[0038] The branch extraction network prediction module is used to determine at least one face feature parameter to be extracted from the plurality of first face feature parameters, and input the face feature parameter to be extracted into the corresponding branch extraction network to predict the second face feature parameter corresponding to the face feature parameter to be extracted.

[0039] The model training module is used to jointly train the neural network model based on the first loss corresponding to the first face feature parameters and the second loss corresponding to the second face feature parameters to obtain a trained neural network model, which includes a trained parameter prediction model and a trained branch extraction network.

[0040] The first loss is obtained based on the ground truth value corresponding to the training set of the two-dimensional face images and the first face feature parameter; the second loss includes the loss between the ground truth value corresponding to the training set of the two-dimensional face images and the second face feature parameter, as well as the loss between the face feature parameter to be extracted and the second face feature parameter.

[0041] The present invention also provides a three-dimensional face reconstruction device, comprising:

[0042] The image acquisition module is used to acquire the two-dimensional face image to be reconstructed;

[0043] The parameter prediction module is used to input the two-dimensional face image to be reconstructed into a pre-trained parameter prediction model to predict multiple face feature parameters.

[0044] The face reconstruction module is used to input the multiple face feature parameters into the three-dimensional deformation model to obtain the reconstructed three-dimensional face;

[0045] The pre-trained parameter prediction model is trained based on the training device for the aforementioned parameter prediction model.

[0046] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements a training method for any of the parameter prediction models described above or a three-dimensional face reconstruction method described above.

[0047] The present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements a training method for any of the parameter prediction models described above or a three-dimensional face reconstruction method described above.

[0048] The parameter prediction model training method and 3D face reconstruction method and apparatus provided by this invention improve the accuracy of facial feature parameters by setting up a branch extraction network. The neural network model is jointly trained based on a first loss corresponding to the first facial feature parameters and a second loss corresponding to the second facial feature parameters, thereby obtaining a trained parameter prediction model. The first loss is obtained based on the ground truth value corresponding to the 2D face image training set and the first facial feature parameters; the second loss includes the loss between the ground truth value corresponding to the 2D face image training set and the second facial feature parameters, as well as the loss between the facial feature parameters to be extracted and the second facial feature parameters. Therefore, the trained neural network model can accurately predict facial feature parameters using only the parameter prediction model, and the subsequent branch extraction network can be directly omitted, thus improving accuracy without increasing computational load. Attached Figure Description

[0049] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0050] Figure 1 This is a flowchart illustrating the training method of the parameter prediction model provided in this embodiment of the invention.

[0051] Figure 2 This is a schematic diagram of the neural network model structure provided in an embodiment of the present invention;

[0052] Figure 3 This is a schematic diagram of the branching refining network provided in an embodiment of the present invention;

[0053] Figure 4 This is a flowchart illustrating the three-dimensional face reconstruction method provided in an embodiment of the present invention;

[0054] Figure 5This is a structural block diagram of the training device for the parameter prediction model provided in this embodiment of the invention;

[0055] Figure 6 This is a structural block diagram of the three-dimensional face reconstruction device provided in the embodiments of the present invention;

[0056] Figure 7 This is a schematic diagram of the physical structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0057] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0058] Figure 1 This is a flowchart illustrating the training method of the parameter prediction model provided in this embodiment of the invention; as shown below. Figure 1 As shown, a method for training a parameter prediction model includes the following steps:

[0059] S101: Obtain a training set of two-dimensional face images and input them into a pre-built neural network model.

[0060] In this step, the 2D face image training set can be a dataset commonly used in the fields of face alignment and face reconstruction, such as the 300W-LP and AFLW2000-3D datasets. It can also be a dataset collected and annotated by the user. This invention does not limit this.

[0061] In addition, the training set of two-dimensional face images includes two-dimensional face images and corresponding three-dimensional face deformation parameters (that is, the ground truth values ​​corresponding to the two-dimensional face images).

[0062] Figure 2 This is a schematic diagram of the neural network model structure provided in an embodiment of the present invention; as shown below. Figure 2 As shown, the pre-built neural network model includes a parameter prediction model and at least one branch extraction network; the parameter prediction model includes a backbone network and multiple detection branch networks.

[0063] In this process, after the facial feature parameters are predicted from the 2D face image by the parametric prediction model, they are further input into the branch refinement network. The branch refinement network further refines and optimizes the facial feature parameters predicted by the parametric prediction model. The number of branch refinement networks corresponds to the number of facial feature parameters for which the prediction accuracy needs to be improved.

[0064] In this embodiment, since the trained parameter prediction model needs to be applied to autonomous vehicles, the lightweight convolutional neural network MobileNetv2 is selected as the parameter prediction model. Other lightweight convolutional neural networks can also be used, such as MobileNetv3, SqueezeNet, ShuffleNet, etc. Alternatively, it can be based on VGGNeT, GoogLeNet, ResNet, etc., and the specific structure of the parameter prediction model can be determined according to user requirements. This invention does not limit this approach.

[0065] S102, facial features are extracted from the two-dimensional face image training set through the backbone network, and multiple first face feature parameters are predicted based on the facial features using the multiple detection branch networks.

[0066] The first facial feature parameters include parameters that affect facial deformation, such as facial pose, shape, expression, skin, and scene lighting.

[0067] Specifically, the backbone network extracts facial features from the 2D face image training set and inputs them into multiple detection branch networks. Each detection branch network can predict the corresponding first facial feature parameter based on the facial features. For example, the detection branch network used to predict facial pose parameters predicts the corresponding facial pose parameter as the first facial feature parameter based on the facial features extracted by the backbone network; the detection branch network used to predict facial shape parameters predicts the corresponding facial shape parameter as the first facial feature parameter based on the facial features extracted by the backbone network, and so on.

[0068] S103, determine at least one face feature parameter to be refined from the plurality of first face feature parameters, and input the face feature parameter to be refined into the corresponding branch refinement network to predict the second face feature parameter corresponding to the face feature parameter to be refined.

[0069] In this step, although the first facial feature parameters include multiple parameters such as facial pose, shape, expression, skin, and scene lighting, not all parameters need to be refined and optimized in different scenarios. For example, in the in-cabin facial reconstruction of autonomous vehicles, scene lighting parameters do not need to be refined and optimized according to the facial reconstruction requirements. Or, if the user only wants to improve the accuracy of one or a few parameters, then only the first facial feature parameters selected by the user can be refined and optimized. In this case, the selected first facial feature parameters are the facial feature parameters to be refined.

[0070] After determining the facial feature parameters to be extracted, the branch extraction network is used to refine and optimize the facial feature parameters to be extracted, thereby obtaining the second facial feature parameters output by the branch extraction network. The second facial feature parameters correspond to the facial feature parameters to be extracted.

[0071] S104, the neural network model is jointly trained according to the first loss corresponding to the first face feature parameters and the second loss corresponding to the second face feature parameters to obtain a trained neural network model, which includes a trained parameter prediction model and a trained branch extraction network.

[0072] The first loss is obtained based on the ground truth value corresponding to the training set of the two-dimensional face images and the first face feature parameter; the second loss includes the loss between the ground truth value corresponding to the training set of the two-dimensional face images and the second face feature parameter, as well as the loss between the face feature parameter to be extracted and the second face feature parameter.

[0073] like Figure 2 As shown, the first face feature parameter output by the parameter prediction model corresponds to a loss value (i.e., the first loss), and the second face feature parameter output by the branch extraction network corresponds to a loss value (i.e., the second loss).

[0074] The first loss is obtained based on the ground truth value corresponding to the 2D face image training set and the first face feature parameters. Based on this first loss, and through backpropagation, the first face feature parameters can gradually approach the ground truth value corresponding to the training set during the training process. In other words, the first loss can be used to train the parameters in the parameter prediction model.

[0075] The second loss includes the loss between the ground truth value corresponding to the 2D face image training set and the second face feature parameters, and the loss between the face feature parameters to be extracted and the second face feature parameters. The loss between the ground truth value corresponding to the training set and the second face feature parameters allows the second face feature parameters to gradually approach the ground truth value during training; while the loss between the face feature parameters to be extracted and the second face feature parameters allows the face feature parameters to gradually approach the second face feature parameters during training. In other words, not only can the parameters in the branch extraction network be trained using the second loss, but the parameter prediction model can also be further trained. This means that the final face feature parameters to be extracted are the optimized face feature parameters, and the parameter prediction model can be directly applied to real-world scenarios.

[0076] This invention provides a training method for a parameter prediction model, which improves the accuracy of facial feature parameters by setting up a branch extraction network. The method involves jointly training the neural network model based on a first loss corresponding to the first facial feature parameters and a second loss corresponding to the second facial feature parameters, thereby obtaining a trained neural network model. The first loss is obtained based on the ground truth value corresponding to the two-dimensional facial image training set and the first facial feature parameters; the second loss includes the loss between the ground truth value corresponding to the two-dimensional facial image training set and the second facial feature parameters, as well as the loss between the facial feature parameters to be extracted and the second facial feature parameters. Therefore, the trained neural network model can accurately predict facial feature parameters using only the parameter prediction model, and the subsequent branch extraction network can be directly omitted, thus improving accuracy without increasing computational load.

[0077] Furthermore, after extracting facial features from the two-dimensional face image training set through the backbone network and predicting multiple first facial feature parameters based on the facial features using the multiple detection branch networks, the method further includes:

[0078] Based on the first facial feature parameters, the reference facial coordinates, and the reference facial parameters, the first facial feature parameters are converted into facial key points.

[0079] Specifically, based on the reference face coordinates, the corresponding deformed face parameters are obtained according to the first face feature parameters output by the parameter prediction model and the reference face parameters corresponding to the first face feature parameters.

[0080] Accordingly, the branching network includes m sequentially connected convolutional layers, and there is a skip connection between the m-th convolutional layer and one of the previous m-1 convolutional layers.

[0081] The step of inputting the facial feature parameters to be extracted into the corresponding branch extraction network to predict the second facial feature parameters corresponding to the facial feature parameters to be extracted includes:

[0082] The facial feature parameters to be refined are extracted through the first m-1 convolutional layers to obtain the feature information output by the m-1th convolutional layer.

[0083] The feature information output by the (m-1)th convolutional layer, the feature information output by the convolutional layer connected to the mth convolutional layer via skip connections, and the facial key points are input into the mth convolutional layer to output the second facial feature parameters.

[0084] For example, such as Figure 3As shown, the branch extraction network consists of four convolutional layers, each a one-dimensional convolution. The input to the first convolutional layer is the facial feature parameters to be extracted. The input to the second convolutional layer is the output of the first convolutional layer, and so on. The input to the third convolutional layer is the output of the second convolutional layer. The input to the fourth convolutional layer includes the output of the third convolutional layer, the output of the second convolutional layer, and the facial key points obtained after transforming the facial feature parameters output by the parameter prediction model. The second and fourth convolutional layers are connected by skip connections. The fourth convolutional layer extracts the second facial feature parameters based on the aforementioned three inputs.

[0085] It should be noted that the output of each convolutional layer is feature information, but the size of each feature information varies, ranging from the initial feature information with a lot of redundant feature information to the final more abstract feature information.

[0086] In addition, the branching refining network in this embodiment includes four one-dimensional convolutional layers. In other embodiments of the present invention, the number of one-dimensional convolutional layers may be adjusted according to the specific refining effect, and the present invention does not limit this.

[0087] This invention provides a training method for a parameter prediction model. By setting skip connections in the branch extraction network, the branch extraction network can combine information from shallow and deep networks, thereby improving the parameter prediction effect.

[0088] Further, the second loss includes the loss between the ground truth corresponding to the two-dimensional face image training set and the second face feature parameters, as well as the loss between the face feature parameters to be extracted and the second face feature parameters, including:

[0089] The loss between the ground truth and the second face feature parameters corresponding to the training set of the two-dimensional face image and the loss between the face feature parameters to be extracted and the second face feature parameters are calculated using the mean square error calculation method.

[0090] Specifically, the second loss includes sub-loss 1 and sub-loss 2, wherein sub-loss 1 is the loss between the ground truth corresponding to the two-dimensional face image training set and the second face feature parameters, and sub-loss 2 is the loss between the face feature parameters to be extracted and the second face feature parameters.

[0091] The formula for calculating sub-loss 1 (i.e., loss_21) is: In the formula, n is the number of facial key points, which is usually 68; target_21_i is the ground truth value corresponding to the i-th facial key point; and pred_21_i is the second facial feature parameter corresponding to the i-th facial key point.

[0092] Similarly, the formula for calculating sub-loss 2 (i.e., loss_22) is: In the formula, n is the number of facial key points, target_22_i is the facial feature parameter to be extracted corresponding to the i-th facial key point, and pred_22_i is the second facial feature parameter corresponding to the i-th facial key point.

[0093] In summary, the second loss is loss = loss_21 + loss_22.

[0094] It should be noted that the calculation process for the first loss is similar to that for sub-loss 1 and sub-loss 2. Specifically, the formula for calculating the first loss (i.e., loss_1) is as follows: In the formula, n is the number of facial key points, target_1_i is the ground truth value corresponding to the i-th facial key point, and pred_1_i is the first facial feature parameter corresponding to the i-th facial key point.

[0095] Furthermore, the first facial feature parameters include at least pose parameters, shape parameters, and expression parameters.

[0096] The reference facial parameters include reference expression parameters and reference shape parameters.

[0097] Accordingly, the step of converting the first facial feature parameters into facial key points based on the first facial feature parameters, the reference facial coordinates, and the reference facial parameters includes:

[0098] Obtain the product between the expression parameter and the baseline expression parameter, and the product between the shape parameter and the baseline shape parameter, respectively.

[0099] Based on the baseline face coordinates, the sum of the product between the expression parameters and the baseline expression parameters and the product between the shape parameters and the baseline shape parameters is obtained as the deformation parameters of the expression and shape.

[0100] Obtain the product between the deformation parameters and pose parameters of the expression and shape, and use it as the facial keypoint under the first facial feature parameters.

[0101] Specifically, facial landmarks:

[0102] Landmarks=pose_param*(s+exp_base*exp_param+s_base*shape_param)

[0103] In the formula, exp_param is the expression parameter, pose_param is the pose parameter, and shape_param is the shape parameter, s is the reference face coordinates, exp_base is the reference expression parameter, and s_base is the reference shape parameter.

[0104] In this embodiment, since the application scenario is set in the face reconstruction inside the cabin of an autonomous vehicle, the first face feature parameters include pose parameters, shape parameters, and expression parameters. In other embodiments of the present invention, the first face feature parameters also include skin parameters, scene lighting parameters, and other feature parameters that affect face reconstruction.

[0105] Furthermore, the number of the branch extraction networks is the same as the number of the facial feature parameters to be extracted, and the branch extraction networks correspond one-to-one with the facial feature parameters to be extracted.

[0106] Accordingly, determining at least one facial feature parameter to be extracted from the plurality of first facial feature parameters includes:

[0107] The facial feature parameters to be extracted are determined from the first facial feature parameters according to the preset accuracy improvement target.

[0108] The facial feature parameters to be extracted are input into the corresponding branch extraction network.

[0109] The preset accuracy improvement target refers to the facial feature parameters that the user needs to improve. For example, if the user only wants to improve the accuracy of the pose parameters, then only the pose parameters are followed by a branch refinement network. The remaining shape parameters, expression parameters, and other first facial feature parameters do not need to be refined and optimized. The corresponding remaining first facial feature parameters only have a first loss and are not involved in the process of training and updating based on the second loss network parameters.

[0110] For example, such as Figure 2 As shown, each first facial feature parameter is determined as the facial feature parameter to be extracted. Each facial feature parameter to be extracted is followed by a branch extraction network. By training and updating the parameter prediction model and all branch extraction networks, the trained parameter prediction model can achieve good accuracy on other feature parameters that affect face reconstruction, such as pose parameters, shape parameters, expression parameters, skin parameters, and scene lighting parameters.

[0111] The present invention provides a method for training a parameter prediction model. By setting branches to refine the network according to user needs, the trained parameter prediction model can have better prediction accuracy on the facial feature parameters selected by the user. The entire training process of the parameter prediction model is more personalized and can reduce the need for corresponding network parameter updates.

[0112] Figure 4 This is a flowchart illustrating the three-dimensional face reconstruction method provided in an embodiment of the present invention; as shown below. Figure 4 As shown, a three-dimensional face reconstruction method includes the following steps:

[0113] S401, Obtain the 2D face image to be reconstructed.

[0114] In this step, the 2D face image to be reconstructed is the 2D image that needs to be reconstructed into a 3D image. It can be the test set corresponding to the training set of the aforementioned 2D face images, or it can be obtained through on-site collection, or it can be obtained from the Internet.

[0115] S402, the two-dimensional face image to be reconstructed is input into a pre-trained parameter prediction model to predict multiple face feature parameters.

[0116] In this step, the pre-trained parameter prediction model only includes the parameter prediction model mentioned above, and does not include the branch extraction network. Since the parameter prediction model is trained based on the first loss and the second loss, the facial feature parameters output by the final parameter prediction model are close to the facial feature parameters output by the branch extraction network. Therefore, in practical applications, high parameter prediction accuracy can be obtained by using only the parameter prediction model, and removing the branch extraction network achieves the effect of no increase in computation.

[0117] S403, input the multiple facial feature parameters into the three-dimensional deformation model to obtain the reconstructed three-dimensional face.

[0118] In this step, the 3D deformation model can obtain the reconstructed 3D face by predicting the face parameters output by the trained parameter prediction model.

[0119] The three-dimensional face reconstruction method provided in this invention uses a trained parameter prediction model to predict facial feature parameters, and then uses a three-dimensional deformation model to complete the three-dimensional face reconstruction, thereby improving the discriminative power of the reconstructed face.

[0120] The training apparatus for the parameter prediction model provided by the present invention will be described below. The training apparatus for the parameter prediction model described below can be referred to in correspondence with the training method for the parameter prediction model described above.

[0121] Figure 5 This is a structural block diagram of the training device for the parameter prediction model provided in this embodiment of the invention, as shown below. Figure 5 As shown, a training device for a parameter prediction model includes:

[0122] The training set acquisition and input module 501 is used to acquire a two-dimensional face image training set and input it into a pre-built neural network model.

[0123] The pre-built neural network model includes a parameter prediction model and at least one branch extraction network; the parameter prediction model includes a backbone network and multiple detection branch networks.

[0124] In this module, the 2D face image training set can be a dataset commonly used in the fields of face alignment and face reconstruction, such as the 300W-LP and AFLW2000-3D datasets. It can also be a dataset collected and annotated by the user. This invention does not limit this.

[0125] In addition, the training set of two-dimensional face images includes two-dimensional face images and corresponding three-dimensional face deformation parameters (that is, the ground truth values ​​corresponding to the two-dimensional face images).

[0126] In this process, after the facial feature parameters are predicted from the 2D face image by the parametric prediction model, they are further input into the branch refinement network. The branch refinement network further refines and optimizes the facial feature parameters predicted by the parametric prediction model. The number of branch refinement networks corresponds to the number of facial feature parameters for which the prediction accuracy needs to be improved.

[0127] In this embodiment, since the trained parameter prediction model needs to be applied to autonomous vehicles, the lightweight convolutional neural network MobileNetv2 is selected as the parameter prediction model. Other lightweight convolutional neural networks can also be used, such as MobileNetv3, SqueezeNet, ShuffleNet, etc. Alternatively, it can be based on VGGNeT, GoogLeNet, ResNet, etc., and the specific structure of the parameter prediction model can be determined according to user requirements. This invention does not limit this approach.

[0128] The parameter prediction model prediction module 502 extracts facial features from the two-dimensional face image training set through the backbone network, and uses the multiple detection branch networks to predict multiple first facial feature parameters based on the facial features.

[0129] The first facial feature parameters include parameters that affect facial deformation, such as facial pose, shape, expression, skin, and scene lighting.

[0130] Specifically, the backbone network extracts facial features from the 2D face image training set and inputs them into multiple detection branch networks. Each detection branch network can predict the corresponding first facial feature parameter based on the facial features. For example, the detection branch network used to predict facial pose parameters predicts the corresponding facial pose parameter as the first facial feature parameter based on the facial features extracted by the backbone network; the detection branch network used to predict facial shape parameters predicts the corresponding facial shape parameter as the first facial feature parameter based on the facial features extracted by the backbone network, and so on.

[0131] The branch extraction network prediction module 503 is used to determine at least one face feature parameter to be extracted from the plurality of first face feature parameters, and input the face feature parameter to be extracted into the corresponding branch extraction network to predict the second face feature parameter corresponding to the face feature parameter to be extracted.

[0132] In this module, although the first facial feature parameters include multiple parameters such as facial pose, shape, expression, skin, and scene lighting, not all parameters need to be refined and optimized in different scenarios. For example, in the in-cabin facial reconstruction of autonomous vehicles, scene lighting parameters do not need to be refined and optimized according to the facial reconstruction requirements. Alternatively, if the user only wants to improve the accuracy of one or a few parameters, then only the first facial feature parameter selected by the user can be refined and optimized. In this case, the selected first facial feature parameter is the facial feature parameter to be refined.

[0133] After determining the facial feature parameters to be extracted, the branch extraction network is used to refine and optimize the facial feature parameters to be extracted, thereby obtaining the second facial feature parameters output by the branch extraction network. The second facial feature parameters correspond to the facial feature parameters to be extracted.

[0134] The model training module 504 is used to jointly train the neural network model based on the first loss corresponding to the first face feature parameters and the second loss corresponding to the second face feature parameters to obtain a trained neural network model, which includes a trained parameter prediction model and a trained branch extraction network.

[0135] The first loss is obtained based on the ground truth value corresponding to the training set of the two-dimensional face images and the first face feature parameter; the second loss includes the loss between the ground truth value corresponding to the training set of the two-dimensional face images and the second face feature parameter, as well as the loss between the face feature parameter to be extracted and the second face feature parameter.

[0136] like Figure 2 As shown, the first face feature parameter output by the parameter prediction model corresponds to a loss value (i.e., the first loss), and the second face feature parameter output by the branch extraction network corresponds to a loss value (i.e., the second loss).

[0137] The first loss is constructed based on the ground truth value corresponding to the 2D face image training set and the first face feature parameters. Based on this first loss, and through backpropagation, the first face feature parameters can gradually approach the ground truth value corresponding to the training set during the training process. In other words, the first loss can be used to train the parameters in the parameter prediction model.

[0138] The second loss includes the loss between the ground truth value corresponding to the 2D face image training set and the second face feature parameters, and the loss between the face feature parameters to be extracted and the second face feature parameters. The loss between the ground truth value corresponding to the training set and the second face feature parameters allows the second face feature parameters to gradually approach the ground truth value during training; while the loss between the face feature parameters to be extracted and the second face feature parameters allows the face feature parameters to gradually approach the second face feature parameters during training. In other words, not only can the parameters in the branch extraction network be trained using the second loss, but the parameter prediction model can also be further trained. This means that the final face feature parameters to be extracted are the optimized face feature parameters, and the parameter prediction model can be directly applied to real-world scenarios.

[0139] The parameter prediction model training device provided in this embodiment of the invention improves the accuracy of facial feature parameters by setting up a branch extraction network. It jointly trains the neural network model based on a first loss corresponding to the first facial feature parameters and a second loss corresponding to the second facial feature parameters, thereby obtaining a trained neural network model. The first loss is obtained based on the ground truth value corresponding to the two-dimensional facial image training set and the first facial feature parameters; the second loss includes the loss between the ground truth value corresponding to the two-dimensional facial image training set and the second facial feature parameters, as well as the loss between the facial feature parameters to be extracted and the second facial feature parameters. Therefore, the trained neural network model can accurately predict facial feature parameters using only the parameter prediction model, and the subsequent branch extraction network can be directly omitted, thus improving accuracy without increasing computational load.

[0140] The three-dimensional face reconstruction device provided by the present invention will be described below. The three-dimensional face reconstruction device described below can be referred to in correspondence with the three-dimensional face reconstruction method described above.

[0141] Figure 6 This is a structural block diagram of the three-dimensional face reconstruction device provided in an embodiment of the present invention, as shown below. Figure 6 As shown, a three-dimensional face reconstruction device includes:

[0142] Image acquisition module 601 is used to acquire two-dimensional face images to be reconstructed.

[0143] In this module, the 2D face image to be reconstructed is the 2D image that needs to be reconstructed into a 3D image. It can be the test set corresponding to the above-mentioned 2D face image training set, or it can be obtained through on-site collection, or obtained from the Internet.

[0144] The parameter prediction module 602 is used to input the two-dimensional face image to be reconstructed into a pre-trained parameter prediction model to predict multiple face feature parameters.

[0145] In this module, the pre-trained parameter prediction model only includes the parameter prediction model mentioned above, and does not include the branch extraction network. Since the parameter prediction model is trained based on the first loss and the second loss, the facial feature parameters output by the final parameter prediction model are close to the facial feature parameters output by the branch extraction network. Therefore, in practical applications, high parameter prediction accuracy can be obtained by using only the parameter prediction model, and removing the branch extraction network achieves the effect of no increase in computation.

[0146] The face reconstruction module 603 is used to input the multiple face feature parameters into the three-dimensional deformation model to obtain the reconstructed three-dimensional face.

[0147] In this module, the 3D deformation model can obtain the reconstructed 3D face by predicting the face parameters output by the trained parameter prediction model.

[0148] The three-dimensional face reconstruction device provided in this embodiment of the invention predicts face feature parameters by using a trained parameter prediction model, and then completes three-dimensional face reconstruction by using a three-dimensional deformation model, thereby improving the discriminative power of the reconstructed face.

[0149] Figure 7 This is a schematic diagram of the physical structure of an electronic device provided in an embodiment of the present invention, such as... Figure 7As shown, the electronic device may include: a processor 710, a communication interface 720, a memory 730, and a communication bus 740, wherein the processor 710, the communication interface 720, and the memory 730 communicate with each other through the communication bus 740. The processor 710 can call logical instructions in the memory 730 to execute a training method for a parameter prediction model. The training method for the parameter prediction model includes: acquiring a two-dimensional face image training set and inputting it into a pre-constructed neural network model; wherein the pre-constructed neural network model includes a parameter prediction model and at least one branch extraction network; the parameter prediction model includes a backbone network and multiple detection branch networks; extracting face features from the two-dimensional face image training set through the backbone network, and predicting multiple first face feature parameters based on the face features using the multiple detection branch networks; determining at least one face feature parameter to be extracted from the multiple first face feature parameters, and assigning the face feature parameter to be extracted to the target face feature network. Facial feature parameters are input into the corresponding branch extraction network to predict second facial feature parameters corresponding to the facial feature parameters to be extracted. The neural network model is jointly trained based on the first loss corresponding to the first facial feature parameters and the second loss corresponding to the second facial feature parameters to obtain a trained neural network model. The trained neural network model includes a trained parameter prediction model and a trained branch extraction network. The first loss is obtained based on the ground truth value corresponding to the two-dimensional facial image training set and the first facial feature parameters. The second loss includes the loss between the ground truth value corresponding to the two-dimensional facial image training set and the second facial feature parameters, as well as the loss between the facial feature parameters to be extracted and the second facial feature parameters.

[0150] Alternatively, it can be used to perform a three-dimensional face reconstruction method, which includes: acquiring a two-dimensional face image to be reconstructed; inputting the two-dimensional face image to be reconstructed into a pre-trained parameter prediction model to predict multiple face feature parameters; inputting the multiple face feature parameters into a three-dimensional deformation model to obtain the reconstructed three-dimensional face; wherein the pre-trained parameter prediction model is trained based on the above-mentioned parameter prediction model training method.

[0151] Furthermore, the logical instructions in the aforementioned memory 730 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0152] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements a training method for the parameter prediction model provided by the above-described method. The training method for the parameter prediction model includes: acquiring a two-dimensional face image training set and inputting it into a pre-constructed neural network model; wherein the pre-constructed neural network model includes a parameter prediction model and at least one branch extraction network; the parameter prediction model includes a backbone network and multiple detection branch networks; extracting face features from the two-dimensional face image training set through the backbone network, and predicting multiple first face feature parameters based on the face features using the multiple detection branch networks; and determining at least one from the multiple first face feature parameters. A face feature parameter to be extracted is input into a corresponding branch extraction network to predict a second face feature parameter corresponding to the face feature parameter to be extracted. The neural network model is then jointly trained based on a first loss corresponding to the first face feature parameter and a second loss corresponding to the second face feature parameter to obtain a trained neural network model. This trained neural network model includes a trained parameter prediction model and a trained branch extraction network. The first loss is obtained based on the ground truth value corresponding to the two-dimensional face image training set and the first face feature parameter. The second loss includes the loss between the ground truth value corresponding to the two-dimensional face image training set and the second face feature parameter, as well as the loss between the face feature parameter to be extracted and the second face feature parameter.

[0153] Alternatively, it can be used to perform a three-dimensional face reconstruction method, which includes: acquiring a two-dimensional face image to be reconstructed; inputting the two-dimensional face image to be reconstructed into a pre-trained parameter prediction model to predict multiple face feature parameters; inputting the multiple face feature parameters into a three-dimensional deformation model to obtain the reconstructed three-dimensional face; wherein the pre-trained parameter prediction model is trained based on the above-mentioned parameter prediction model training method.

[0154] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0155] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of various embodiments or some parts of embodiments.

[0156] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A training method for a parameter prediction model, characterized in that, include: A training set of two-dimensional face images is acquired and input into a pre-built neural network model; wherein the pre-built neural network model includes a parameter prediction model and at least one branch extraction network; the parameter prediction model includes a backbone network and multiple detection branch networks; Facial features are extracted from the two-dimensional face image training set through the backbone network, and multiple first facial feature parameters are predicted based on the facial features using the multiple detection branch networks. At least one face feature parameter to be extracted is determined from the plurality of first face feature parameters, and the face feature parameter to be extracted is input into the corresponding branch extraction network to predict the second face feature parameter corresponding to the face feature parameter to be extracted. The neural network model is jointly trained based on the first loss corresponding to the first facial feature parameters and the second loss corresponding to the second facial feature parameters to obtain a trained neural network model, which includes a trained parameter prediction model and a trained branch extraction network. The first loss is obtained based on the ground truth value corresponding to the training set of the two-dimensional face images and the first face feature parameter; the second loss includes the loss between the ground truth value corresponding to the training set of the two-dimensional face images and the second face feature parameter, as well as the loss between the face feature parameter to be extracted and the second face feature parameter.

2. The training method for the parameter prediction model according to claim 1, characterized in that, After extracting facial features from the two-dimensional face image training set through the backbone network and predicting multiple first facial feature parameters based on the facial features using the multiple detection branch networks, the method further includes: Based on the first facial feature parameters, the reference facial coordinates, and the reference facial parameters, the first facial feature parameters are converted into facial key points; Accordingly, the branching network includes m sequentially connected convolutional layers, and there is a skip connection between the m-th convolutional layer and one of the previous m-1 convolutional layers; The step of inputting the facial feature parameters to be extracted into the corresponding branch extraction network to predict the second facial feature parameters corresponding to the facial feature parameters to be extracted includes: The facial feature parameters to be refined are extracted through the first m-1 convolutional layers to obtain the feature information output by the m-1th convolutional layer; The feature information output by the (m-1)th convolutional layer, the feature information output by the convolutional layer connected to the mth convolutional layer via skip connections, and the facial key points are input into the mth convolutional layer to output the second facial feature parameters.

3. The training method for the parameter prediction model according to claim 1, characterized in that, The second loss includes the loss between the ground truth value corresponding to the two-dimensional face image training set and the second face feature parameters, as well as the loss between the face feature parameters to be extracted and the second face feature parameters, including: The loss between the ground truth and the second face feature parameters corresponding to the training set of the two-dimensional face image and the loss between the face feature parameters to be extracted and the second face feature parameters are calculated using the mean square error calculation method.

4. The training method for the parameter prediction model according to claim 2, characterized in that, The first facial feature parameters include at least pose parameters, shape parameters, and expression parameters; The reference facial parameters include reference expression parameters and reference shape parameters; Accordingly, the step of converting the first facial feature parameters into facial key points based on the first facial feature parameters, the reference facial coordinates, and the reference facial parameters includes: Obtain the product between the expression parameter and the baseline expression parameter, and the product between the shape parameter and the baseline shape parameter, respectively; Based on the baseline face coordinates, the sum of the product between the expression parameters and the baseline expression parameters and the product between the shape parameters and the baseline shape parameters is obtained as the deformation parameters of the expression and shape. Obtain the product between the deformation parameters and pose parameters of the expression and shape, and use it as the facial keypoint under the first facial feature parameters.

5. The training method for the parameter prediction model according to any one of claims 1-4, characterized in that, The number of branch extraction networks is the same as the number of facial feature parameters to be extracted, and the branch extraction networks correspond one-to-one with the facial feature parameters to be extracted; Accordingly, determining at least one facial feature parameter to be extracted from the plurality of first facial feature parameters includes: The facial feature parameters to be extracted are determined from the first facial feature parameters according to the preset accuracy improvement target; The facial feature parameters to be extracted are input into the corresponding branch extraction network.

6. A three-dimensional face reconstruction method, characterized in that, include: Obtain the 2D face image to be reconstructed; The two-dimensional face image to be reconstructed is input into a pre-trained parameter prediction model to predict multiple face feature parameters. The multiple facial feature parameters are input into a three-dimensional deformation model to obtain the reconstructed three-dimensional face; The pre-trained parameter prediction model is trained based on the parameter prediction model training method described in any one of claims 1-5.

7. A training device for a parameter prediction model, characterized in that, include: The training set acquisition and input module is used to acquire a two-dimensional face image training set and input it into a pre-built neural network model; wherein, the pre-built neural network model includes a parameter prediction model and at least one branch extraction network; the parameter prediction model includes a backbone network and multiple detection branch networks; The parameter prediction model prediction module is used to extract facial features from the two-dimensional face image training set through the backbone network, and use the multiple detection branch networks to predict multiple first facial feature parameters based on the facial features. The branch extraction network prediction module is used to determine at least one face feature parameter to be extracted from the plurality of first face feature parameters, and input the face feature parameter to be extracted into the corresponding branch extraction network to predict the second face feature parameter corresponding to the face feature parameter to be extracted. The model training module is used to jointly train the neural network model based on the first loss corresponding to the first face feature parameters and the second loss corresponding to the second face feature parameters to obtain a trained neural network model, which includes a trained parameter prediction model and a trained branch extraction network. The first loss is obtained based on the ground truth value corresponding to the training set of the two-dimensional face images and the first face feature parameter; the second loss includes the loss between the ground truth value corresponding to the training set of the two-dimensional face images and the second face feature parameter, as well as the loss between the face feature parameter to be extracted and the second face feature parameter.

8. A three-dimensional face reconstruction device, characterized in that, include: The image acquisition module is used to acquire the two-dimensional face image to be reconstructed; The parameter prediction module is used to input the two-dimensional face image to be reconstructed into a pre-trained parameter prediction model to predict multiple face feature parameters. The face reconstruction module is used to input the multiple face feature parameters into the three-dimensional deformation model to obtain the reconstructed three-dimensional face; The pre-trained parameter prediction model is trained based on the training device for the parameter prediction model as described in claim 7.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the training method of the parameter prediction model as described in any one of claims 1 to 5 or the three-dimensional face reconstruction method as described in claim 6.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the training method of the parameter prediction model as described in any one of claims 1 to 5 or the three-dimensional face reconstruction method as described in claim 6.

Images