Model training method, 3D face reconstruction method, device and storage medium

Abstract

The present application relates to a model training method, a 3D face reconstruction method, a device and a storage medium. Since the method introduces compensation parameters in the process of training the model, the training loss that can be obtained is more accurate than that without compensation parameters, and it is easier for the neural network model to converge. And since the PCA basis is not introduced in the process of training the model, when the trained model is used for 3D face reconstruction, there is no problem of low reconstruction accuracy due to the influence of the accuracy and dimension of the PCA basis.

Description

Technical Field

[0001] The present application relates to the field of computers, and in particular to a model training method, a 3D face reconstruction method, a device and a storage medium. Background Art

[0002] With the development of society and the advancement of science and technology, three-dimensional (3D) face reconstruction technology has gradually become a popular technology in the field of computer vision. 3D face reconstruction technology is to reconstruct a 3D face model based on a single or multiple face images of the individual being tested.

[0003] Current optimization-based and learning-based 3D face reconstruction methods both rely on the PCA basis. On the one hand, they are affected by the accuracy of the PCA basis. On the other hand, the PCA basis dimension is generally low and its expression ability is limited. Even if a neural network model is used to learn the basis coefficients of the face, the obtained basis coefficients lose a large part of the information, resulting in low final reconstruction accuracy. Summary of the invention

[0004] The present application provides a model training method, a 3D face reconstruction method, a device and a storage medium to solve the problem of low reconstruction accuracy due to the influence of the accuracy and dimension of the PCA basis.

[0005] In a first aspect, a model training method is provided, comprising:

[0006] Obtaining a two-dimensional face image sample used for executing the model training method for the Nth time;

[0007] Based on the 3D face prediction module in the neural network model, predict the three-dimensional coordinate values ​​of N three-dimensional coordinate points corresponding to the two-dimensional face image sample; and based on the 3D face compensation module in the neural network model, predict the compensation parameters of the two-dimensional face image sample; the 3D face compensation module includes a mapping module and / or an illumination module, when the 3D face compensation module is the mapping module, the compensation parameter is the face mapping, and when the 3D face compensation module is the illumination module, the compensation parameter is the illumination coefficient;

[0008] Rendering the compensation parameter and the three-dimensional coordinate values ​​of the N three-dimensional coordinate points to obtain a predicted two-dimensional face image corresponding to the two-dimensional face image sample;

[0009] Calculating a training loss based on the predicted two-dimensional face image and the two-dimensional face image sample; and determining the training stage of the neural network model based on the Nth time, the training stage representing an optimization method for the 3D face compensation module and the illumination module;

[0010] According to the optimization method characterized by the training stage, the parameters of the neural network model are optimized using the training loss, N=N+1 is updated, and the model training method is executed for the N+1th time until the neural network model converges.

[0011] Optionally, the 3D face compensation module includes a mapping module and an illumination module;

[0012] Rendering the compensation parameter and the three-dimensional coordinate values ​​of the N three-dimensional coordinate points to obtain a predicted two-dimensional face image corresponding to the two-dimensional face image sample, including:

[0013] Obtain a de-lighted image of the two-dimensional face image sample; render the face map, the illumination coefficient, and the three-dimensional coordinate values ​​of the N three-dimensional coordinate points, and predict a rendered image corresponding to the two-dimensional face image sample; render the face map and the three-dimensional coordinate values ​​of the N three-dimensional coordinate points, and predict a non-illuminated rendered image corresponding to the two-dimensional face image sample;

[0014] The de-lighted image, the rendered image and the un-illuminated rendered image are used as the predicted two-dimensional face image.

[0015] Optionally, calculating the training loss based on the predicted two-dimensional face image and the two-dimensional face image sample includes:

[0016] Based on the two-dimensional face image sample and the rendered image, respectively calculate the loss of the rendered image, the face recognition model feature loss, the classification model feature loss, the lip contour loss, and the eye contour loss; based on the de-lighted image and the unilluminated rendered image, calculate the loss of the unilluminated rendered image; based on the two-dimensional face image sample and the unilluminated rendered image, calculate the loss of facial key points;

[0017] The training loss is calculated based on the loss of the rendered image, the face recognition model feature loss, the classification model feature loss, the lip contour loss, the eye contour loss, the loss of the unilluminated rendered image, and the loss of the face key points.

[0018] Optionally, based on the two-dimensional face image sample and the rendered image, respectively calculating the loss of the rendered image, the face recognition model feature loss, the classification model feature loss, the lip contour loss, and the eye contour loss, including:

[0019] Calculating the L2 distance between the two-dimensional face image sample and the rendered image to obtain a first calculation result; using the first calculation result as the loss of the rendered image;

[0020] Extracting the first face recognition model feature of the two-dimensional face image sample and the second face recognition model feature of the rendered image; calculating the L2 distance between the first face recognition model feature and the second face recognition model feature to obtain a second settlement result; and using the second calculation result as the face recognition model feature loss;

[0021] Obtain the features of the two-dimensional face image sample extracted by the classification model and the rendered image at the i-th layer of the classification model; calculate the L2 distance between the features of the two-dimensional face image sample at the i-th layer and the features of the rendered image at the i-th layer to obtain a third calculation result; calculate the product value of the third calculation result and the weight coefficient of the i-th layer to obtain the feature loss of the i-th layer; based on the feature loss of the i-th layer, calculate the sum of the feature losses of M layers, and use the sum of the feature losses of the M layers as the feature loss of the classification model; the classification model has an M-layer structure, 1≤i≤M;

[0022] Obtaining a first coordinate of the lip contour in the two-dimensional face image sample and a second coordinate of the lip contour in the rendered image; calculating a first contour distance error, a first contour curvature error, and a first contour radial error between the lip contour in the two-dimensional face image sample and the lip contour in the rendered image based on the first coordinate and the second coordinate; calculating the sum of the first contour distance error, the first contour curvature error, and the first contour radial error, and using the sum as the lip contour loss;

[0023] Obtain the third coordinate of the eye contour in the two-dimensional face image sample and the fourth coordinate of the eye contour in the rendered image; based on the third coordinate and the fourth coordinate, respectively calculate the second contour distance error, the second contour curvature error and the second contour radial error of the eye contour in the two-dimensional face image sample and the eye contour in the rendered image; calculate the sum of the second contour distance error, the second contour curvature error and the second contour radial error, and use the sum as the eye contour loss.

[0024] Optionally, calculating the loss of the unlit rendered image based on the matte image and the unlit rendered image includes:

[0025] Calculating an L2 distance between the matte image and the unlit rendered image to obtain a fourth calculation result;

[0026] The fourth calculation result is used as the loss of the unlit rendered image.

[0027] Optionally, calculating the loss of facial key points based on the two-dimensional facial image sample and the unilluminated rendering image includes:

[0028] Obtaining fifth coordinates of key facial points in the two-dimensional facial image sample and sixth coordinates of key facial points in the unilluminated rendering image;

[0029] Calculating the L2 distance between the fifth coordinate and the sixth coordinate to obtain a fifth calculation result;

[0030] The fifth calculation result is used as the loss of the facial key point.

[0031] Optionally, based on a 3D face prediction module in a neural network model, predicting three-dimensional coordinate values ​​of N three-dimensional coordinate points corresponding to the two-dimensional face image sample comprises:

[0032] Using the encoder in the 3D face prediction module, predict a first expression of the two-dimensional face image sample in a two-dimensional space;

[0033] Converting the first expression through a decoder in the 3D face prediction module to obtain a three-dimensional expression of the two-dimensional face image sample in a three-dimensional space;

[0034] The three-dimensional coordinate value of each point in the three-dimensional expression is obtained, and the three-dimensional coordinate value of each point in the three-dimensional expression is used as the three-dimensional coordinate value of the N points.

[0035] In a second aspect, a 3D face reconstruction method is provided, comprising:

[0036] Obtain a two-dimensional face image;

[0037] Using the 3D face prediction module in the neural network model trained by claim 1, predicting the three-dimensional coordinate values ​​of N three-dimensional coordinate points of the two-dimensional face image in the three-dimensional space;

[0038] Based on the three-dimensional coordinate values ​​of the N three-dimensional coordinate points, a three-dimensional face image corresponding to the two-dimensional face image is generated.

[0039] In a third aspect, a model training device is provided, comprising:

[0040] A first acquisition module, used to acquire a two-dimensional face image sample used for executing the model training method for the Nth time;

[0041] A first prediction module is used to predict the three-dimensional coordinate values ​​of N three-dimensional coordinate points corresponding to the two-dimensional face image sample based on the 3D face prediction module in the neural network model; and predict the compensation parameters of the two-dimensional face image sample based on the 3D face compensation module in the neural network model; the 3D face compensation module includes a mapping module and / or an illumination module, when the 3D face compensation module is the mapping module, the compensation parameter is the face mapping, when the 3D face compensation module is the illumination module, the compensation parameter is the illumination coefficient;

[0042] A rendering module, used for rendering the compensation parameter and the three-dimensional coordinate values ​​of the N three-dimensional coordinate points to obtain a predicted two-dimensional face image corresponding to the two-dimensional face image sample;

[0043] A calculation module, configured to calculate a training loss based on the predicted two-dimensional face image and the two-dimensional face image sample; and, based on the N-th determination of the training stage of the neural network model, the training stage characterizing an optimization method for the 3D face compensation module and the illumination module;

[0044] An optimization module is used to optimize the parameters of the neural network model using the training loss according to the optimization method characterized by the training stage, update N=N+1, and execute the model training method for the N+1th time until the neural network model converges.

[0045] In a fourth aspect, a 3D face reconstruction device is provided, comprising:

[0046] A second acquisition module is used to acquire a two-dimensional face image;

[0047] A second prediction module, configured to use the 3D face prediction module in the neural network model trained by claim 1 to predict the three-dimensional coordinate values ​​of N three-dimensional coordinate points of the two-dimensional face image in the three-dimensional space;

[0048] A generating module is used to generate a three-dimensional face image corresponding to the two-dimensional face image based on the three-dimensional coordinate values ​​of the N three-dimensional coordinate points.

[0049] In a fifth aspect, an electronic device is provided, comprising: a processor, a memory and a communication bus, wherein the processor and the memory communicate with each other via the communication bus;

[0050] The memory is used to store computer programs;

[0051] The processor is used to execute the program stored in the memory to implement the model training method described in the first aspect or the 3D face reconstruction method described in the second aspect.

[0052] In a sixth aspect, a computer-readable storage medium is provided, storing a computer program, wherein when the computer program is executed by a processor, the model training method described in the first aspect or the 3D face reconstruction method described in the second aspect is implemented.

[0053] The above technical solution provided by the embodiment of the present application has the following advantages over the prior art: In the technical solution provided by the embodiment of the present application, since compensation parameters are introduced in the process of training the model, the training loss that can be obtained is more accurate than that without compensation parameters, and it is easier for the neural network model to converge. And since the PCA basis is not introduced in the process of training the model, when the trained model is used for 3D face reconstruction, there is no problem of low reconstruction accuracy due to the influence of the accuracy and dimension of the PCA basis. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0055] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0056] Figure 1 This is a flow chart of the model training method in the embodiment of the present application;

[0057] Figure 2 A schematic diagram of a principle of a model training device in an embodiment of the present application;

[0058] Figure 3 This is another schematic diagram of the principle of the model training device in the embodiment of the present application;

[0059] Figure 4 Schematic diagram of the process of 3D face reconstruction method in the embodiment of the present application;

[0060] Figure 5 This is a schematic diagram of the structure of the model training device in the embodiment of the present application;

[0061] Figure 6 This is a schematic diagram of the structure of a 3D face reconstruction device in an embodiment of the present application;

[0062] Figure 7 Schematic diagram of the structure of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION

[0063] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0064] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0065] In order to solve the problem of low reconstruction accuracy, relevant technologies have also proposed using a network model to perform 3D reconstruction of the target face, thereby effectively simplifying the reconstruction process and improving the upper limit of reconstruction accuracy.

[0066] However, when using this solution for training, since there is only a monocular image of the target face, there are fewer loss constraints that can be performed, such as facial key points, facial depth (RGBD), etc., which makes it difficult for the model to converge.

[0067] In order to solve this problem, the present application embodiment provides a model training method, such as Figure 1 As shown, the method can be applied to any electronic device;

[0068] The electronic device described in the embodiments of the present application may include a terminal or a server, which is not limited in the embodiments of the present application. The terminal includes various handheld devices with wireless communication functions, vehicle-mounted devices, wearable devices (such as smart watches, smart bracelets, pedometers, etc.) or computing devices.

[0069] like Figure 1 As shown, the method may include the following steps:

[0070] Step 101: Obtain a two-dimensional face image sample used for executing the model training method for the Nth time.

[0071] Step 102: based on the 3D face prediction module in the neural network model, predict the three-dimensional coordinate values ​​of N three-dimensional coordinate points corresponding to the two-dimensional face image samples; and based on the 3D face compensation module in the neural network model, predict the compensation parameters of the two-dimensional face image samples; the 3D face compensation module includes a mapping module and / or an illumination module, when the 3D face compensation module is a mapping module, the compensation parameter is a face mapping, when the 3D face compensation module is an illumination module, the compensation parameter is an illumination coefficient.

[0072] It should be understood that in practical applications, the two-dimensional face image sample is a two-dimensional plane image. The horizontal direction of the plane image is U, and the vertical direction is V. Through this two-dimensional UV coordinate system, any pixel on the two-dimensional face image sample can be located.

[0073] At present, although neural network models such as CNN have shown very strong expressive power in image-related tasks, when applying neural network models in 3D reconstruction solutions, it is necessary to use the PCA basis to give full play to the capabilities of the neural network model. However, due to the accuracy and dimension of the PCA basis, the accuracy of the 3D face predicted by the neural network model is not high. Therefore, when predicting the three-dimensional coordinate values ​​of N three-dimensional coordinate points with the help of the 3D face prediction module, this embodiment introduces a mapping relationship indicating the three-dimensional space to the UV space to give full play to the advantages of the 3D face prediction module in three-dimensional image reconstruction.

[0074] When a 3D face prediction module is used to predict the three-dimensional coordinate value of a two-dimensional face image sample, an encoder in the 3D face prediction module is used to predict a first expression of the two-dimensional face image sample in a two-dimensional space; the first expression is converted by a decoder in the 3D face prediction module to obtain a three-dimensional expression of the two-dimensional face image sample in a three-dimensional space; the three-dimensional coordinate value of each point in the three-dimensional expression is obtained, and the three-dimensional coordinate value of each point in the three-dimensional expression is used as the three-dimensional coordinate value of N points.

[0075] In this embodiment, taking the 3D face compensation module including the mapping module and the illumination module as an example, the prediction process of the face mapping and the illumination coefficient can be expressed by the formula:

[0076] tex=TexNet(I)(1)

[0077] light=E light (I)(2)

[0078] Among them, tex is the face map, TexNet is the map module, I is the two-dimensional face image sample, light is the illumination coefficient, E light For the lighting module.

[0079] It should be understood that the face map here refers to the texture information of the face surface. The illumination coefficient is used to reflect the brightness and darkness in the two-dimensional face image sample.

[0080] Step 103: Render the compensation parameters and the three-dimensional coordinate values ​​of the N three-dimensional coordinate points to obtain a predicted two-dimensional face image corresponding to the two-dimensional face image sample.

[0081] In the application, when the 3D face compensation module includes a mapping module and an illumination module, the compensation parameters include a face mapping and an illumination coefficient. In this case, the predicted two-dimensional face image includes a de-lighted image, a rendered image, and a non-illuminated rendered image.

[0082] Among them, a pre-trained de-lighting model can be used to remove the illumination of the two-dimensional face image sample to obtain a de-lighting image of the two-dimensional face image sample. A differentiable renderer can be used to render the face map, the illumination coefficient, and the three-dimensional coordinate values ​​of N three-dimensional coordinate points to predict the rendered image of the two-dimensional face image sample; and a differentiable renderer can be used to render the face map and the three-dimensional coordinate values ​​of N three-dimensional coordinate points to predict the unilluminated rendered image corresponding to the two-dimensional face image sample.

[0083] Accordingly, when predicting the rendered image, the prediction process of the differentiable renderer can be expressed as:

[0084]

[0085] in, is the rendered image, Π is the differentiable renderer, M pose are the coordinate values ​​of N three-dimensional coordinate points, tex is the face map, and light is the illumination coefficient.

[0086] When predicting an unlit rendered image, the prediction process of the differentiable renderer can be expressed as:

[0087]

[0088] in, is the unlit rendered image, Π is the differentiable renderer, M pose are the coordinate values ​​of N three-dimensional coordinate points, and tex is the face map.

[0089] Step 104, calculating the training loss based on the predicted two-dimensional face image and the two-dimensional face image sample; and determining the training stage of the neural network model for the Nth time, the training stage representing the optimization method of the 3D face compensation module and the illumination module.

[0090] In this embodiment, since the compensation parameter is introduced, the loss constraint that can be obtained is greatly increased compared to when there is no compensation parameter.

[0091] Still taking the compensation parameters including face map and illumination coefficient as an example, the training loss calculated based on the predicted two-dimensional face image samples and the two-dimensional face image samples can be calculated by the following loss:

[0092] The loss of rendered images, face recognition model feature loss, classification model feature loss, lip contour loss, eye contour loss, loss of unlit rendered images, loss of face key points, texture smoothness loss, texture symmetry loss, and lighting regularization loss are calculated.

[0093] Among them, the loss of rendered images, face recognition model feature loss, classification model feature loss, lip contour loss and eye contour loss are calculated respectively from the two-dimensional face image samples and the rendered images; the loss of unlit rendered images is calculated from the de-illuminated images and the unlit rendered images; the loss of facial key points is calculated from the two-dimensional face image samples and the unlit rendered images.

[0094] The calculation process of the above losses is described below:

[0095] About the loss of rendered images:

[0096] In an optional embodiment, the L2 distance between the two-dimensional face image sample and the rendered image is calculated to obtain a first calculation result; and the first calculation result is used as the loss of the rendered image.

[0097] The formula used to calculate the loss of the rendered image is:

[0098]

[0099] Among them, loss pho is the loss of the rendered image, I is a two-dimensional face image sample, To render the image.

[0100] About the feature loss of face recognition model:

[0101] In an optional embodiment, a first face recognition model feature of a two-dimensional face image sample and a second face recognition model feature of a rendered image are extracted; the L2 distance between the first face recognition model feature and the second face recognition model feature is calculated to obtain a second settlement result; and the second calculation result is used as the face recognition model feature loss.

[0102] The formula used to calculate the feature loss of the face recognition model is:

[0103]

[0104] Among them, loss id is the feature loss of the face recognition model, F face(·) is the face recognition model feature, I is a two-dimensional face image sample, To render the image.

[0105] About classification model feature loss:

[0106] In an optional embodiment, the features of the two-dimensional face image samples extracted by the classification model and the rendered image at the i-th layer of the classification model are obtained; the L2 distance between the features of the two-dimensional face image samples at the i-th layer and the features of the rendered image at the i-th layer is calculated to obtain a third calculation result; the product value of the third calculation result and the weight coefficient of the i-th layer is calculated to obtain the feature loss of the i-th layer; based on the feature loss of the i-th layer, the sum of the feature losses of the M layers is calculated, and the sum of the feature losses of the M layers is used as the feature loss of the classification model; the classification model has an M-layer structure, 1≤i≤M.

[0107] The formula used to calculate the feature loss of the classification model is:

[0108]

[0109] Among them, loss vgg is the classification model feature loss, The i-th layer feature extracted by the classification model, w i is the weight of the corresponding layer feature.

[0110] About lip contour loss:

[0111] In an optional embodiment, a first coordinate of the lip contour in a two-dimensional face image sample and a second coordinate of the lip contour in a rendered image are obtained; based on the first coordinate and the second coordinate, a first contour distance error, a first contour curvature error and a first contour radial error of the lip contour in the two-dimensional face image sample and the lip contour in the rendered image are respectively calculated; the sum of the first contour distance error, the first contour curvature error and the first contour radial error is calculated, and the sum result is used as the lip contour loss.

[0112] The formula used to calculate the lip contour loss is:

[0113]

[0114] Among them, loss lip is the lip contour loss, lip is the first coordinate of the lip contour in the two-dimensional face image sample, is the second coordinate of the lip outline in the rendered image.

[0115] About eye contour:

[0116] In an optional embodiment, the third coordinate of the eye contour in the two-dimensional face image sample and the fourth coordinate of the eye contour in the rendered image are obtained; based on the third coordinate and the fourth coordinate, the second contour distance error, the second contour curvature error and the second contour radial error of the eye contour in the two-dimensional face image sample and the eye contour in the rendered image are calculated respectively; the sum of the second contour distance error, the second contour curvature error and the second contour radial error is calculated, and the sum result is used as the eye contour loss.

[0117] The formula used to calculate the eye contour is:

[0118]

[0119] Among them, loss eye is the eye contour loss, eye is the third coordinate of the eye contour in the two-dimensional face image sample, The fourth coordinate of the eye outline in the rendered image.

[0120] Regarding the loss of unlit rendered images:

[0121] In an optional embodiment, the L2 distance between the matte image and the unlit rendered image is calculated to obtain a fourth calculation result; and the fourth calculation result is used as the loss of the unlit rendered image.

[0122] The formula used to calculate the loss of the unlit rendered image is:

[0123]

[0124] Among them, loss alb is the loss of the unlit rendered image, I alb To deglaze the image, Renders the image for unlit conditions.

[0125] About the loss of facial key points:

[0126] In an optional embodiment, the fifth coordinate of the facial key point in the two-dimensional facial image sample and the sixth coordinate of the facial key point in the unilluminated rendering image are obtained; the L2 distance between the fifth coordinate and the sixth coordinate is calculated to obtain a fifth calculation result; and the fifth calculation result is used as the loss of the facial key point.

[0127] The formula used to calculate the loss of facial key points is:

[0128]

[0129] Among them, loss lm is the loss of facial key points, lm is the fifth coordinate of the facial key points in the two-dimensional face image sample, The sixth coordinate of the facial landmark in the unlit rendered image.

[0130] In this embodiment, in order to further optimize the training loss of the neural network, the loss constraints used to calculate the training loss may also include texture smoothness loss, texture symmetry loss, and illumination regularization loss.

[0131] The formula used to calculate the texture smoothing loss is:

[0132] loss Usmo =L smo (U id )+L smo (U exp )(12)

[0133] Among them, loss Usmo is the texture smoothing loss, U id and U exp M id and M exp The corresponding three-dimensional coordinates, about M id and M exp See also Figure 2 as well as Figure 3 Explained in.

[0134]

[0135] The formula used to calculate the texture symmetry loss is:

[0136] loss texflip =||tex-flip(tex)||1(13)

[0137] Among them, loss texflip is the map symmetry loss, tex is the face map, and flip(·) is the horizontal flip operation.

[0138] About illumination regularization loss:

[0139] The formula used to calculate the illumination regularization loss is:

[0140] loss reg =L reg (light)(14)

[0141] Among them, loss reg is the illumination regularization loss, and light is the illumination coefficient.

[0142] In this embodiment, the training of the neural network model is divided into three stages: initial training, mid-training and late training. Different stages correspond to different training orders. Therefore, the current training stage can be determined based on the current training order. In the early training stage, the neural network model is completely randomly initialized, and the outputs of each module in the neural network model are relatively messy and have no reasonable meaning. These modules will be trained separately. In the mid-training stage, the neural network model has initially converged, and the outputs of each module have met the specific meaning. These modules will be updated alternately. In the late training stage, the modules of the model have converged relatively well, and all modules will be updated together.

[0143] Step 105: According to the optimization method characterized by the training phase, the parameters of the neural network model are optimized using the training loss, N=N+1 is updated, and the N+1th model training method is executed until the neural network model converges.

[0144] In the technical solution provided by the embodiment of the present application, since compensation parameters are introduced in the process of training the model, the training loss that can be obtained is more accurate than that without compensation parameters, and it is easier for the neural network model to converge. In addition, since the PCA basis is not introduced in the process of training the model, when the trained model is used for 3D face reconstruction, there is no problem of low reconstruction accuracy due to the influence of the accuracy and dimension of the PCA basis.

[0145] For ease of understanding, Figure 2 as well as Figure 3 The principle of the embodiment of the present application is described, wherein Figure 2 This is a schematic diagram of the principle when face mapping and illumination coefficient are not introduced. Figure 3 Schematic diagram of the principle after introducing face mapping and illumination coefficient, where I is a two-dimensional face image, E_id is an identity encoder, E_exp is an expression encoder, and E_pose is a posture encoder; f_id is an identity feature, f_exp is an expression feature, and f_pose is a posture feature; D_id is an identity decoder, and D_exp is an expression decoder; M_id is the three-dimensional coordinate value of the three-dimensional coordinate point of the neutral expression in a neutral posture, M_exp is the three-dimensional coordinate value of the three-dimensional coordinate point of the expression corresponding to I in a neutral posture, and M_pose is the three-dimensional coordinate value of the three-dimensional coordinate point of the posture expression corresponding to I; TexNet is a mapping module, E_light is an illumination module, tex is a face map, light is an illumination coefficient, and Renderer is a differentiable renderer; I' is a 2D image rendered by M_pose, tex, and light using a differentiable renderer, and the training goal is to minimize the error between I' and I.

[0146] Based on the same concept, the present application embodiment provides a 3D face reconstruction method, such as Figure 4 As shown, the method may include the following steps:

[0147] Step 401, obtaining a two-dimensional face image;

[0148] Step 402: using a 3D face prediction module in a neural network model trained by a model training method to predict the three-dimensional coordinate values ​​of N three-dimensional coordinate points of a two-dimensional face image in a three-dimensional space;

[0149] Step 403: Generate a three-dimensional face image corresponding to the two-dimensional face image based on the three-dimensional coordinate values ​​of the N three-dimensional coordinate points.

[0150] Based on the same concept, a model training device is provided in the embodiment of the present application. The specific implementation of the device can be found in the description of the method embodiment part, and the repeated parts will not be repeated. Figure 5 As shown, the device mainly includes:

[0151] A first acquisition module 501 is used to acquire a two-dimensional face image sample used for executing the model training method for the Nth time;

[0152] The first prediction module 502 is used to predict the three-dimensional coordinate values ​​of N three-dimensional coordinate points corresponding to the two-dimensional face image samples based on the 3D face prediction module in the neural network model; and predict the compensation parameters of the two-dimensional face image samples based on the 3D face compensation module in the neural network model; the 3D face compensation module includes a mapping module and / or an illumination module, when the 3D face compensation module is a mapping module, the compensation parameter is a face mapping, when the 3D face compensation module is an illumination module, the compensation parameter is an illumination coefficient;

[0153] A rendering module 503 is used to render the compensation parameters and the three-dimensional coordinate values ​​of the N three-dimensional coordinate points to obtain a predicted two-dimensional face image corresponding to the two-dimensional face image sample;

[0154] A calculation module 504 is used to calculate the training loss based on the predicted two-dimensional face image and the two-dimensional face image sample; and, based on the Nth determination of the training stage of the neural network model, the training stage represents the optimization method of the 3D face compensation module and the illumination module;

[0155] The optimization module 505 is used to optimize the parameters of the neural network model using the training loss according to the optimization method characterized by the training stage, update N=N+1, and execute the N+1th model training method until the neural network model converges.

[0156] Optionally, the 3D face compensation module includes a mapping module and an illumination module;

[0157] The rendering module 503 is used for:

[0158] Obtain a de-lighted image of a two-dimensional face image sample; render the face map, the illumination coefficient, and the three-dimensional coordinate values ​​of N three-dimensional coordinate points, and predict the rendered image corresponding to the two-dimensional face image sample; render the face map and the three-dimensional coordinate values ​​of N three-dimensional coordinate points, and predict the unilluminated rendered image corresponding to the two-dimensional face image sample;

[0159] The de-lighted image, the rendered image and the un-illuminated rendered image are used as predicted two-dimensional face images.

[0160] The rendering module 503 is used for:

[0161] Based on the 2D face image samples and the rendered image, the loss of the rendered image, the feature loss of the face recognition model, the feature loss of the classification model, the lip contour loss, and the eye contour loss are calculated respectively; based on the de-lighted image and the unlit rendered image, the loss of the unlit rendered image is calculated; based on the 2D face image samples and the unlit rendered image, the loss of the facial key points is calculated;

[0162] The training loss is calculated based on the loss of the rendered image, the feature loss of the face recognition model, the feature loss of the classification model, the lip contour loss, the eye contour loss, the loss of the unlit rendered image, and the loss of the face key points.

[0163] The rendering module 503 is used for:

[0164] Calculate the L2 distance between the two-dimensional face image sample and the rendered image to obtain a first calculation result; use the first calculation result as the loss of the rendered image;

[0165] Extracting the first face recognition model feature of the two-dimensional face image sample and the second face recognition model feature of the rendered image; calculating the L2 distance between the first face recognition model feature and the second face recognition model feature to obtain a second settlement result; using the second calculation result as the face recognition model feature loss;

[0166] Obtain the features of the two-dimensional face image samples extracted by the classification model and the rendered image at the i-th layer of the classification model; calculate the L2 distance between the features of the two-dimensional face image samples at the i-th layer and the features of the rendered image at the i-th layer to obtain a third calculation result; calculate the product value of the third calculation result and the weight coefficient of the i-th layer to obtain the feature loss of the i-th layer; based on the feature loss of the i-th layer, calculate the sum of the feature losses of the M layers, and use the sum of the feature losses of the M layers as the feature loss of the classification model; the classification model has an M-layer structure, 1≤i≤M;

[0167] Obtaining a first coordinate of a lip contour in a two-dimensional face image sample and a second coordinate of a lip contour in a rendered image; calculating a first contour distance error, a first contour curvature error, and a first contour radial error of the lip contour in the two-dimensional face image sample and the lip contour in the rendered image based on the first coordinate and the second coordinate; calculating the sum of the first contour distance error, the first contour curvature error, and the first contour radial error, and using the sum as the lip contour loss;

[0168] Obtain the third coordinate of the eye contour in the two-dimensional face image sample and the fourth coordinate of the eye contour in the rendered image; based on the third coordinate and the fourth coordinate, respectively calculate the second contour distance error, the second contour curvature error and the second contour radial error of the eye contour in the two-dimensional face image sample and the eye contour in the rendered image; calculate the sum of the second contour distance error, the second contour curvature error and the second contour radial error, and use the sum as the eye contour loss.

[0169] The rendering module 503 is used for:

[0170] Calculate the L2 distance between the matte image and the unlit rendered image to obtain a fourth calculation result;

[0171] The fourth calculation result is used as the loss of the unlit rendered image.

[0172] The rendering module 503 is used for:

[0173] Obtain the fifth coordinate of the facial key point in the two-dimensional facial image sample and the sixth coordinate of the facial key point in the unilluminated rendering image;

[0174] Calculate the L2 distance between the fifth coordinate and the sixth coordinate to obtain a fifth calculation result;

[0175] The fifth calculation result is used as the loss of the face key point.

[0176] The first prediction module 502 is used to:

[0177] Using the encoder in the 3D face prediction module, the first expression of the two-dimensional face image sample in the two-dimensional space is predicted;

[0178] Converting the first expression through a decoder in a 3D face prediction module to obtain a three-dimensional expression of the two-dimensional face image sample in a three-dimensional space;

[0179] The three-dimensional coordinate value of each point in the three-dimensional expression is obtained, and the three-dimensional coordinate value of each point in the three-dimensional expression is used as the three-dimensional coordinate value of N points.

[0180] Based on the same concept, a 3D face reconstruction device is provided in the embodiment of the present application. The specific implementation of the device can be found in the description of the method embodiment part, and the repeated parts will not be repeated. Figure 6 As shown, the device mainly includes:

[0181] The second acquisition module 601 is used to acquire a two-dimensional face image;

[0182] The second prediction module 602 is used to use the 3D face prediction module in the neural network model trained by the model training method to predict the three-dimensional coordinate values ​​of N three-dimensional coordinate points of the two-dimensional face image in the three-dimensional space;

[0183] The generating module 603 is used to generate a three-dimensional face image corresponding to the two-dimensional face image based on the three-dimensional coordinate values ​​of the N three-dimensional coordinate points.

[0184] Based on the same concept, an electronic device is also provided in the embodiment of the present application, such as Figure 7 As shown, the electronic device mainly includes: a processor 701, a memory 702 and a communication bus 703, wherein the processor 701 and the memory 702 communicate with each other through the communication bus 703. The memory 702 stores a program executable by the processor 701, and the processor 701 executes the program stored in the memory 702 to implement the following steps:

[0185] Obtain a two-dimensional face image sample used in the N-th execution of the model training method; based on the 3D face prediction module in the neural network model, predict the three-dimensional coordinate values ​​of N three-dimensional coordinate points corresponding to the two-dimensional face image sample; and based on the 3D face compensation module in the neural network model, predict the compensation parameters of the two-dimensional face image sample; the 3D face compensation module includes a mapping module and / or an illumination module, when the 3D face compensation module is a mapping module, the compensation parameter is a face mapping, and when the 3D face compensation module is an illumination module, the compensation parameter is an illumination coefficient; render the compensation parameter and the three-dimensional coordinate values ​​of the N three-dimensional coordinate points to obtain a predicted two-dimensional face image corresponding to the two-dimensional face image sample; calculate the training loss based on the predicted two-dimensional face image and the two-dimensional face image sample; and, based on the N-th determination of the training stage of the neural network model, the training stage represents the optimization method for the 3D face compensation module and the illumination module; according to the optimization method represented by the training stage, the parameters of the neural network model are optimized using the training loss, N=N+1 is updated, and the N+1-th model training method is executed until the neural network model converges;

[0186] or,

[0187] Acquire a two-dimensional face image; use the 3D face prediction module in the neural network model trained by the above-mentioned model training method to predict the three-dimensional coordinate values ​​of N three-dimensional coordinate points of the two-dimensional face image in three-dimensional space; based on the three-dimensional coordinate values ​​of the N three-dimensional coordinate points, generate a three-dimensional face image corresponding to the two-dimensional face image.

[0188] The communication bus 703 mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The communication bus 703 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 7 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0189] The memory 702 may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Alternatively, the memory may also be at least one storage device located away from the processor 701.

[0190] The above-mentioned processor 701 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc., and can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.

[0191] In another embodiment of the present application, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program runs on a computer, the computer executes the model training method or 3D face reconstruction method described in the above embodiments.

[0192] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instruction is loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. The computer instruction can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instruction is transmitted from a website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server, a data center, etc. that contains one or more available media integration. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape, etc.), an optical medium (e.g., a DVD) or a semiconductor medium (e.g., a solid-state hard disk), etc.

[0193] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0194] The foregoing is merely a specific embodiment of the present invention, which enables those skilled in the art to understand or implement the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A model training method, characterized in that: include: Obtaining a two-dimensional face image sample used for executing the model training method for the Nth time; Based on the 3D face prediction module in the neural network model, predict the three-dimensional coordinate values ​​of N three-dimensional coordinate points corresponding to the two-dimensional face image sample; and based on the 3D face compensation module in the neural network model, predict the compensation parameters of the two-dimensional face image sample; the 3D face compensation module includes a mapping module and / or an illumination module, when the 3D face compensation module is the mapping module, the compensation parameter is the face mapping, and when the 3D face compensation module is the illumination module, the compensation parameter is the illumination coefficient; Rendering the compensation parameter and the three-dimensional coordinate values ​​of the N three-dimensional coordinate points to obtain a predicted two-dimensional face image corresponding to the two-dimensional face image sample; Calculating the training loss based on the predicted two-dimensional face image and the two-dimensional face image sample, including: calculating the loss of the rendered image, the face recognition model feature loss, the classification model feature loss, the lip contour loss and the eye contour loss based on the two-dimensional face image sample and the rendered image corresponding to the two-dimensional face image sample; calculating the loss of the unilluminated rendered image based on the de-lighted image of the two-dimensional face image sample and the unilluminated rendered image corresponding to the two-dimensional face image sample; calculating the loss of facial key points based on the two-dimensional face image sample and the unilluminated rendered image; calculating the training loss based on the loss of the rendered image, the face recognition model feature loss, the classification model feature loss, the lip contour loss, the eye contour loss and the loss of the unilluminated rendered image, the loss of the facial key points, the texture smoothness loss, the texture symmetry loss and the illumination regularization loss; and, based on the Nth determination of the training stage of the neural network model, the training stage represents the optimization method of the 3D face compensation module and the illumination module; According to the optimization method characterized by the training stage, the parameters of the neural network model are optimized using the training loss, N=N+1 is updated, and the model training method is executed for the N+1th time until the neural network model converges.

2. The method according to claim 1, characterized in that The 3D face compensation module includes a mapping module and a lighting module; Rendering the compensation parameter and the three-dimensional coordinate values ​​of the N three-dimensional coordinate points to obtain a predicted two-dimensional face image corresponding to the two-dimensional face image sample, including: Obtain a de-lighted image of the two-dimensional face image sample; render the face map, the illumination coefficient, and the three-dimensional coordinate values ​​of the N three-dimensional coordinate points, and predict a rendered image corresponding to the two-dimensional face image sample; render the face map and the three-dimensional coordinate values ​​of the N three-dimensional coordinate points, and predict a non-illuminated rendered image corresponding to the two-dimensional face image sample; The de-lighted image, the rendered image and the un-illuminated rendered image are used as the predicted two-dimensional face image.

3. The method according to claim 2, characterized in that Based on the two-dimensional face image sample and the rendered image, respectively calculating the loss of the rendered image, the face recognition model feature loss, the classification model feature loss, the lip contour loss, and the eye contour loss, including: Calculating the L2 distance between the two-dimensional face image sample and the rendered image to obtain a first calculation result; using the first calculation result as the loss of the rendered image; Extracting a first face recognition model feature of the two-dimensional face image sample and a second face recognition model feature of the rendered image; calculating an L2 distance between the first face recognition model feature and the second face recognition model feature to obtain a second calculation result; and using the second calculation result as the face recognition model feature loss; Obtain the features of the two-dimensional face image sample extracted by the classification model and the rendered image at the i-th layer of the classification model; calculate the L2 distance between the features of the two-dimensional face image sample at the i-th layer and the features of the rendered image at the i-th layer to obtain a third calculation result; calculate the product value of the third calculation result and the weight coefficient of the i-th layer to obtain the feature loss of the i-th layer; based on the feature loss of the i-th layer, calculate the sum of the feature losses of M layers, and use the sum of the feature losses of the M layers as the feature loss of the classification model; the classification model has an M-layer structure, 1≤i≤M; Obtaining a first coordinate of the lip contour in the two-dimensional face image sample and a second coordinate of the lip contour in the rendered image; calculating a first contour distance error, a first contour curvature error, and a first contour radial error between the lip contour in the two-dimensional face image sample and the lip contour in the rendered image based on the first coordinate and the second coordinate; calculating the sum of the first contour distance error, the first contour curvature error, and the first contour radial error, and using the sum as the lip contour loss; Obtain the third coordinate of the eye contour in the two-dimensional face image sample and the fourth coordinate of the eye contour in the rendered image; based on the third coordinate and the fourth coordinate, respectively calculate the second contour distance error, the second contour curvature error and the second contour radial error of the eye contour in the two-dimensional face image sample and the eye contour in the rendered image; calculate the sum of the second contour distance error, the second contour curvature error and the second contour radial error, and use the sum as the eye contour loss.

4. The method according to claim 2, characterized in that: Calculating the loss of the unlit rendered image based on the de-lighted image and the unlit rendered image includes: Calculating an L2 distance between the matte image and the unlit rendered image to obtain a fourth calculation result; The fourth calculation result is used as the loss of the unlit rendered image.

5. The method according to claim 2, characterized in that: Calculating the loss of facial key points based on the two-dimensional facial image sample and the unilluminated rendering image includes: Obtaining fifth coordinates of key facial points in the two-dimensional facial image sample and sixth coordinates of key facial points in the unilluminated rendering image; Calculating the L2 distance between the fifth coordinate and the sixth coordinate to obtain a fifth calculation result; The fifth calculation result is used as the loss of the facial key point.

6. The method according to any one of claims 1 to 5, characterized in that: Based on the 3D face prediction module in the neural network model, predicting the three-dimensional coordinate values ​​of N three-dimensional coordinate points corresponding to the two-dimensional face image sample includes: Using the encoder in the 3D face prediction module, predict a first expression of the two-dimensional face image sample in a two-dimensional space; Converting the first expression through a decoder in the 3D face prediction module to obtain a three-dimensional expression of the two-dimensional face image sample in a three-dimensional space; The three-dimensional coordinate value of each point in the three-dimensional expression is obtained, and the three-dimensional coordinate value of each point in the three-dimensional expression is used as the three-dimensional coordinate value of the N points.

7. A 3D face reconstruction method, characterized in that: include: Obtain a two-dimensional face image; Using the 3D face prediction module in the neural network model trained by claim 1, predicting the three-dimensional coordinate values ​​of N three-dimensional coordinate points of the two-dimensional face image in the three-dimensional space; Based on the three-dimensional coordinate values ​​of the N three-dimensional coordinate points, a three-dimensional face image corresponding to the two-dimensional face image is generated.

8. A model training device, characterized in that: include: A first acquisition module, used to acquire a two-dimensional face image sample used for executing the model training method for the Nth time; A first prediction module is used to predict the three-dimensional coordinate values ​​of N three-dimensional coordinate points corresponding to the two-dimensional face image sample based on the 3D face prediction module in the neural network model; and predict the compensation parameters of the two-dimensional face image sample based on the 3D face compensation module in the neural network model; the 3D face compensation module includes a mapping module and / or an illumination module, when the 3D face compensation module is the mapping module, the compensation parameter is the face mapping, when the 3D face compensation module is the illumination module, the compensation parameter is the illumination coefficient; A rendering module, used for rendering the compensation parameter and the three-dimensional coordinate values ​​of the N three-dimensional coordinate points to obtain a predicted two-dimensional face image corresponding to the two-dimensional face image sample; A calculation module, for calculating the training loss based on the predicted two-dimensional face image and the two-dimensional face image sample, including: based on the two-dimensional face image sample and the rendered image corresponding to the two-dimensional face image sample, respectively calculating the loss of the rendered image, the face recognition model feature loss, the classification model feature loss, the lip contour loss and the eye contour loss; based on the de-lighted image of the two-dimensional face image sample and the unilluminated rendered image corresponding to the two-dimensional face image sample, calculating the loss of the unilluminated rendered image; based on the two-dimensional face image sample and the unilluminated rendered image, calculating the loss of facial key points; based on the loss of the rendered image, the face recognition model feature loss, the classification model feature loss, the lip contour loss, the eye contour loss and the loss of the unilluminated rendered image, the loss of the facial key points, the texture smoothness loss, the texture symmetry loss and the illumination regularization loss, calculating the training loss; and, based on the Nth determination of the training stage of the neural network model, the training stage represents the optimization method of the 3D face compensation module and the illumination module; An optimization module is used to optimize the parameters of the neural network model using the training loss according to the optimization method characterized by the training stage, update N=N+1, and execute the model training method for the N+1th time until the neural network model converges.

9. A 3D face reconstruction device, characterized in that: include: A second acquisition module is used to acquire a two-dimensional face image; A second prediction module, configured to use the 3D face prediction module in the neural network model trained by claim 1 to predict the three-dimensional coordinate values ​​of N three-dimensional coordinate points of the two-dimensional face image in the three-dimensional space; A generating module is used to generate a three-dimensional face image corresponding to the two-dimensional face image based on the three-dimensional coordinate values ​​of the N three-dimensional coordinate points.

10. An electronic device, characterized in that: include: A processor, a memory and a communication bus, wherein the processor and the memory communicate with each other via the communication bus; The memory is used to store computer programs; The processor is used to execute the program stored in the memory to implement the model training method described in any one of claims 1 to 6 or the 3D face reconstruction method described in claim 7.

11. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the model training method according to any one of claims 1 to 6 or the 3D face reconstruction method according to claim 7 is implemented.

Images