Three-dimensional face reconstruction method, electronic device and computer-readable storage medium

Abstract

The three-dimensional face reconstruction method, electronic device and computer-readable storage medium provided in the embodiments of the present application relate to the field of image processing technology. The embodiments of the present application divide the target face in the two-dimensional face image into a first part that does not need to be corrected and a second part to be corrected. After predicting the three-dimensional point cloud of the target face, the three-dimensional point cloud of the second part of the target face is corrected to obtain the corrected three-dimensional point cloud of the target face. In this way, the above method can correct the second part with a larger movement amplitude when the overall posture of the face predicted in the three-dimensional face prediction network model fits the actual face shape, optimize the fitting effect of the predicted result of the part with a larger movement amplitude with the actual part, and take into account the fitting effect of the predicted overall posture of the face and the local part.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a three-dimensional face reconstruction method, an electronic device, and a computer-readable storage medium. Background Art

[0002] In live broadcast application scenarios, there will be some demand for 3D animation effects, such as rabbit ears, pig masks and clown hats, etc. These 3D animation effects need to be based on 3D face reconstruction. 3D face reconstruction not only includes the positioning of some key points of the face, but also includes the state information of the face at this time, such as the rotation angle information of the face. In addition, 3D face reconstruction may also have some personalized needs in certain scenarios. For example, after a user takes a selfie, a 3D cartoon image with a high degree of similarity to the user is generated based on the user's selfie.

[0003] Existing 3D face reconstruction schemes include those based on deep learning and those based on regression coefficients. The deep learning-based scheme can make the predicted overall face posture fit the actual face shape better, but for parts with large movements (such as the mouth or eyes), the predicted results fit the actual parts poorly. The traditional regression coefficient scheme can better achieve the fit between the predicted results and the actual parts for parts with large movements, but the predicted overall face posture fits the actual face shape poorly. Therefore, how to take into account the fit between the predicted overall face posture and the local parts in the 3D face reconstruction process is a technical problem that technicians in this field urgently need to solve. Summary of the invention

[0004] In order to at least overcome the above-mentioned deficiencies in the prior art, the purpose of the present application is to provide a three-dimensional face reconstruction method, an electronic device and a computer-readable storage medium.

[0005] In a first aspect, an embodiment of the present application provides a three-dimensional face reconstruction method, the method comprising:

[0006] Inputting a frame of two-dimensional face image into a trained three-dimensional face prediction network model to predict facial key points to obtain a three-dimensional point cloud of a target face in the two-dimensional face image, wherein the target face includes a first part that does not need to be corrected and a second part that needs to be corrected;

[0007] Acquire a two-dimensional point cloud of the target face, wherein the two-dimensional point cloud of the first part and the two-dimensional point cloud of the second part are acquired in different ways;

[0008] Based on the three-dimensional point cloud and the two-dimensional point cloud of the target face, the three-dimensional point cloud of the second part of the target face is corrected to obtain a corrected three-dimensional point cloud of the target face.

[0009] In the above scheme, the target face in the two-dimensional face image is divided into a first part that does not need to be corrected and a second part to be corrected. After the three-dimensional point cloud of the target face is predicted, the three-dimensional point cloud of the second part of the target face is corrected to obtain the corrected three-dimensional point cloud of the target face. In this way, the above method can correct the second part with a larger movement amplitude when the overall face posture predicted in the three-dimensional face prediction network model fits the actual face shape, optimize the fit effect between the prediction result of the part with a larger movement amplitude and the actual part, and take into account the fit effect of the predicted overall face posture and the local part.

[0010] In a possible implementation, before the step of inputting a frame of two-dimensional face image into a trained three-dimensional face prediction network model to predict facial key points to obtain a three-dimensional point cloud of a target face in the two-dimensional face image, the method further includes a step of training the three-dimensional face prediction network model, which includes:

[0011] Acquire a training data set, the training data set comprising a plurality of two-dimensional face image samples carrying label parameters, the label parameters comprising face posture label parameters and face deformation label parameters;

[0012] Inputting the two-dimensional face image samples into the deep learning network model to be trained for processing, and outputting three-dimensional face prediction parameters, wherein the three-dimensional face prediction parameters include face posture prediction parameters and face deformation prediction parameters;

[0013] Performing face reconstruction based on the three-dimensional face prediction parameters to obtain a first three-dimensional point cloud projected onto the face in the two-dimensional face image sample, and performing face reconstruction based on the label parameters of the two-dimensional face image sample to obtain a second three-dimensional point cloud projected onto the face in the two-dimensional face image sample;

[0014] The model loss function value of the deep learning network model is calculated by the first three-dimensional point cloud and the second three-dimensional point cloud, and when the model loss function value is greater than or equal to the preset loss function domain value, the model parameters of the deep learning network model are adjusted by back propagation of gradients, and the step of inputting the two-dimensional face image samples into the deep learning network model to be trained for processing in sequence and outputting three-dimensional face prediction parameters is returned to, until the model loss function value of the first three-dimensional point cloud and the second three-dimensional point cloud is less than the preset loss function threshold, and the trained deep learning network model is obtained as the three-dimensional face prediction network model.

[0015] In a possible implementation, the step of inputting a frame of a two-dimensional face image into a trained three-dimensional face prediction network model to perform face key point prediction to obtain a three-dimensional point cloud of a target face in the two-dimensional face image includes:

[0016] Inputting the two-dimensional face image into a trained three-dimensional face prediction network model to output three-dimensional face prediction parameters of a target face in the two-dimensional face image;

[0017] Face reconstruction is performed based on the three-dimensional face prediction parameters of the target face to obtain a three-dimensional point cloud projected onto the target face.

[0018] In a possible implementation, the step of obtaining the two-dimensional point cloud of the target face includes:

[0019] Obtaining a two-dimensional point cloud of a first part of the target face based on the three-dimensional point cloud of the first part of the target face;

[0020] A two-dimensional point cloud detector is used to obtain a two-dimensional point cloud of a second part of the target face, and a two-dimensional point cloud of the target face is obtained from the two-dimensional point cloud of the first part and the two-dimensional point cloud of the second part.

[0021] In a possible implementation, the step of correcting the three-dimensional point cloud of the second part of the target face based on the three-dimensional point cloud and the two-dimensional point cloud of the target face to obtain the corrected three-dimensional point cloud of the target face includes:

[0022] Calculating an error value between a projection of a three-dimensional point cloud corresponding to the target face on a two-dimensional plane and the two-dimensional point cloud;

[0023] Optimizing the error value by adjusting the three-dimensional face prediction parameters of the target face so that the error value meets a preset condition;

[0024] Face reconstruction is performed based on the adjusted three-dimensional face prediction parameters of the target face to obtain a corrected three-dimensional point cloud of the target face.

[0025] In a possible implementation, the face deformation prediction parameter includes an expression coefficient, and the step of optimizing the error value by adjusting the three-dimensional face prediction parameter of the target face so that the error value satisfies a preset condition includes:

[0026] The error value is optimized by adjusting the expression coefficient of the target face and keeping other parameters in the three-dimensional face prediction parameters of the target face except the expression coefficient unchanged, so that the error value reaches a minimum value, wherein the method for optimizing the error value includes gradient descent method, least squares method, non-negative least squares method and L-BFGS algorithm.

[0027] In a possible implementation, the step of performing face reconstruction based on the adjusted three-dimensional face prediction parameters of the target face to obtain a corrected three-dimensional point cloud of the target face includes:

[0028] The expression coefficient corresponding to the time when the error value reaches the minimum value and other parameters of the three-dimensional face prediction parameters of the target face except the expression coefficient form the adjusted three-dimensional face prediction parameters of the target face;

[0029] The adjusted three-dimensional face prediction parameters of the target face are used to perform face reconstruction to obtain a corrected three-dimensional point cloud of the target face.

[0030] In a possible implementation, the face posture prediction parameters include scaling parameters, rotation matrices, and offset matrices, and the face deformation prediction parameters also include shape coefficients. The formula used for face reconstruction is as follows:

[0031]

[0032] Among them, X is the projected 3D point cloud, s is the scaling parameter, P is the orthogonal projection matrix, and R is the rotation matrix. The average face modeled by the PCA 3D model is a known quantity, α i is the PCA basis of shape coefficients, s i is the shape coefficient, m is the dimension of the shape coefficient, β i is the PCA basis of the expression coefficient in the deformation parameter, e i is the expression coefficient, n is the dimension of the expression coefficient, t 2d is the offset matrix.

[0033] In a second aspect, the present application also provides a 3D face reconstruction device, the device comprising:

[0034] A prediction model is used to input a frame of a two-dimensional face image into a trained three-dimensional face prediction network model to predict facial key points to obtain a three-dimensional point cloud of a target face in the two-dimensional face image, wherein the target face includes a first part that does not need to be corrected and a second part that needs to be corrected;

[0035] An acquisition model is used to acquire a two-dimensional point cloud of the target face, wherein the two-dimensional point cloud of the first part and the two-dimensional point cloud of the second part are acquired in different ways;

[0036] The correction model is used to correct the three-dimensional point cloud of the second part of the target face based on the three-dimensional point cloud and the two-dimensional point cloud of the target face to obtain the corrected three-dimensional point cloud of the target face.

[0037] In a third aspect, an embodiment of the present application further provides an electronic device, comprising a processor and a computer-readable storage medium, wherein the processor and the computer-readable storage medium are connected via a bus system, the computer-readable storage medium is used to store programs, instructions or codes, and the processor is used to execute the programs, instructions or codes in the computer-readable storage medium to implement the three-dimensional face reconstruction method in any possible implementation method of the first aspect.

[0038] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores instructions that, when executed, enable a computer to execute the three-dimensional face reconstruction method in the above-mentioned first aspect or any possible implementation of the first aspect.

[0039] Based on any of the above aspects, the three-dimensional face reconstruction method, electronic device and computer-readable storage medium provided in the embodiments of the present application divide the target face in the two-dimensional face image into a first part that does not need to be corrected and a second part to be corrected. After predicting the three-dimensional point cloud of the target face, the three-dimensional point cloud of the second part of the target face is corrected to obtain the corrected three-dimensional point cloud of the target face. In this way, the above method can correct the second part with a larger movement amplitude when the overall face posture predicted in the three-dimensional face prediction network model fits the actual face shape, optimize the fit effect between the predicted result of the part with a larger movement amplitude and the actual part, and take into account the fit effect between the predicted overall face posture and the local part. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0041] Figure 1 A schematic diagram of a flow chart of a three-dimensional face reconstruction method provided in an embodiment of the present application;

[0042] Figure 2 A schematic diagram of the sub-step flow of training a 3D face prediction network model provided in an embodiment of the present application;

[0043] Figure 3 A schematic diagram of the mapping relationship between a three-dimensional point cloud and a two-dimensional point cloud is provided for the embodiment of the present application;

[0044] Figure 4 for Figure 2 Schematic diagram of the sub-step flow chart of step S13;

[0045] Figure 5 A schematic diagram of the bonding effect achieved based on the technical solution provided in the embodiment of the present application;

[0046] Figure 6 A schematic diagram of the functional modules of the three-dimensional face reconstruction device provided in an embodiment of the present application;

[0047] Figure 7 A schematic diagram of a possible structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0048] To make the purpose, technical scheme and advantages of the embodiment of the present application clearer, the technical scheme in the embodiment of the present application will be clearly and completely described below in conjunction with the drawings in the embodiment of the present application. It should be understood that the drawings in the present application only serve the purpose of explanation and description and are not used to limit the scope of protection of the present application. In addition, it should be understood that the schematic drawings are not drawn in real proportion. The flowchart used in this application shows the operations implemented according to some embodiments of the embodiment of the present application. It should be understood that the operations of the flowchart can be implemented out of sequence, and the steps without logical context can be reversed in order or implemented simultaneously. In addition, those skilled in the art can add one or more other operations to the flowchart under the guidance of the content of the present application, or remove one or more operations from the flowchart.

[0049] In addition, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0050] Please refer to Figure 1 , Figure 1 The flowchart of the 3D face reconstruction method provided by the embodiment of the present application is illustrated. The 3D face reconstruction method can be executed by an electronic device (for example, a server). In other embodiments, the order of some steps in the 3D face reconstruction method of the embodiment of the present application can be interchanged according to actual needs, or some steps can be omitted or deleted. Figure 1 The detailed steps of the three-dimensional face reconstruction method are introduced.

[0051] Step S11, input a frame of two-dimensional face image into the trained three-dimensional face prediction network model to predict facial key points to obtain a three-dimensional point cloud of the target face in the two-dimensional face image.

[0052] Among them, the two-dimensional face image can be sampled from the video image, for example, it can be sampled from the live video of the anchor, and the sampling frequency can be set according to actual needs, for example, one frame of two-dimensional face image can be sampled every 5 frames.

[0053] In this step, the 3D face prediction network model can be trained based on the deep learning network model, and the overall facial posture represented by the 3D point cloud of the target face fits well with the actual face shape. The 3D point cloud includes a large number of 3D key points of the face, and each 3D key point of the face can include three-dimensional coordinates. The facial contour represented by the 3D point cloud is the contour of the real face, which will be hidden behind the target face when the angle is large.

[0054] The target face may include a first part that does not need to be corrected and a second part to be corrected, wherein the second part to be corrected refers to a part with a larger range of movement, such as the mouth and eyes; the first part is the other part of the target face excluding the second part, such as the nose and facial contour.

[0055] Step S12, obtaining a two-dimensional point cloud of the target face.

[0056] In this step, the two-dimensional point cloud includes a large number of two-dimensional key points of the face, and each two-dimensional key point of the face includes a two-dimensional coordinate corresponding to the two-dimensional face image. The two-dimensional point cloud of the first part is obtained in a different manner from the two-dimensional point cloud of the second part, wherein the facial contour represented by the two-dimensional point cloud is always located at the facial edge of the target face.

[0057] Step S13, based on the three-dimensional point cloud and the two-dimensional point cloud of the target face, correct the three-dimensional point cloud of the second part of the target face to obtain a corrected three-dimensional point cloud of the target face.

[0058] In this step, a mapping relationship between the three-dimensional point cloud of the target face and the corresponding two-dimensional point cloud can be constructed, and then the three-dimensional point cloud of the target face predicted by the three-dimensional face prediction network model can be corrected through the above mapping relationship to obtain the corrected three-dimensional point cloud of the target face.

[0059] In the embodiment of the present application, based on the above technical solution, the target face in the two-dimensional face image can be divided into a first part that does not need to be corrected and a second part to be corrected. After the three-dimensional point cloud of the target face is predicted, the three-dimensional point cloud of the second part of the target face is corrected to obtain the corrected three-dimensional point cloud of the target face. In this way, when the overall posture of the face predicted in the three-dimensional face prediction network model fits the actual face shape, the second part with a larger movement amplitude can be corrected, and the fitting effect of the predicted result of the part with a larger movement amplitude and the actual part can be optimized, taking into account the fitting effect of the predicted overall posture of the face and the local part.

[0060] Furthermore, before step S11, the 3D face reconstruction method provided in the embodiment of the present application may also include the step of training a 3D face prediction network model, see Figure 2 , Figure 2 The following is a schematic diagram of the sub-step process of training a 3D face prediction network model. Figure 2 This paper introduces the training process of the 3D face prediction network model.

[0061] Sub-step S101, obtaining a training data set.

[0062] In a possible implementation of this embodiment, the training data set may include multiple two-dimensional face image samples carrying label parameters, wherein the label parameters include face posture label parameters and face deformation label parameters. The training data set may be based on an open source database (e.g., ORL database, FG-NET database, LFW database, etc.) and obtained through a traditional regression coefficient method.

[0063] Sub-step S102, sequentially inputting the two-dimensional face image samples into the deep learning network model to be trained for processing, and outputting three-dimensional face prediction parameters.

[0064] In this embodiment, the deep learning network model can adopt the network structure of the 3D face alignment model (3DDense FaceAlignment, 3DDFA). The three-dimensional face prediction parameters may include face posture prediction parameters and face deformation prediction parameters, the face posture label parameters and face posture prediction parameters may include scaling parameters, rotation matrices and offset matrices, and the face deformation label parameters and face deformation prediction parameters may include expression coefficients and shape coefficients.

[0065] Sub-step S103, performing face reconstruction based on the three-dimensional face prediction parameters to obtain a first three-dimensional point cloud projected onto the face in the two-dimensional face image sample, and performing face reconstruction based on the label parameters of the two-dimensional face image sample to obtain a second three-dimensional point cloud projected onto the face in the two-dimensional face image sample.

[0066] By substituting the three-dimensional face prediction parameters into the face reconstruction formula, the first three-dimensional point cloud projected onto the face in the two-dimensional face image sample can be obtained. By substituting the label parameters into the face reconstruction formula, the second three-dimensional point cloud projected onto the face in the two-dimensional face image sample can be obtained. The face reconstruction formula is as follows:

[0067]

[0068] Among them, X is the projected 3D point cloud, s is the scaling parameter, P is the orthogonal projection matrix, and R is the rotation matrix. The average face modeled by the PCA 3D model is a known quantity, α i is the PCA basis of shape coefficients, s i is the shape coefficient, m is the dimension of the shape coefficient, β i is the PCA basis of the expression coefficient in the deformation parameter, e i is the expression coefficient, n is the dimension of the expression coefficient, t 2d is the offset matrix.

[0069] Sub-step S104, calculating the model loss function value of the deep learning network model through the first three-dimensional point cloud and the second three-dimensional point cloud, and determining whether the model loss function value is less than a preset loss function domain value.

[0070] As an example, the Euclidean distance between the first three-dimensional point cloud and the second three-dimensional point cloud can be used as the model loss function value. When the model loss function value is greater than or equal to the preset loss function domain value, sub-step S105 is entered; when the model loss function value is less than the preset loss function domain value, sub-step S106 is entered.

[0071] Sub-step S105, adjusting the model parameters of the deep learning network model by back-propagating the gradient, and then entering the next iterative training process (for example, returning to step S102).

[0072] After the adjustment of the model parameters is completed, return to sub-step S101 until the loss function value of the cloud model of the first three-dimensional point cloud and the second three-dimensional point cloud is less than a preset loss function threshold.

[0073] Sub-step S106, using the trained deep learning network model as the 3D face prediction network model, and ending the training.

[0074] In the embodiment of the present application, the results obtained by the three-dimensional face prediction network model obtained by deep learning network model training are biased towards the global sample average, and the fitting effect of some cases with few occurrences will be poor. When the business has high requirements for memory and time consumption (such as live broadcast scenes), it is impossible to fit some difficult cases under limited parameters. However, the three-dimensional face prediction network model based on deep learning has a good effect on the overall face posture and face shape fitting, because during the training process, if the overall face posture and face shape do not fit, the predicted results will be greatly different from the label parameters, and the returned gradient will also be very large, so the parameters give priority to controlling these cases. However, for parts with large movement amplitudes (for example, large-angle mouth opening cases), the predicted results may have the situation that the mouth does not fit, but it will only affect the key points of the mouth area, and the overall deviation is small.

[0075] Furthermore, in the embodiment of the present application, step S11 can be implemented in the following manner.

[0076] First, the two-dimensional face image is input into the trained three-dimensional face prediction network model to output the three-dimensional face prediction parameters of the target face in the two-dimensional face image.

[0077] Then, face reconstruction is performed based on the three-dimensional face prediction parameters of the target face to obtain a three-dimensional point cloud projected onto the target face.

[0078] Among them, different from the training process of the 3D face prediction network model described above, step S11 may only need to predict the 3D face prediction parameters of the target face, and reconstruct the 3D point cloud projected onto the target face based on the 3D face prediction parameters of the target face.

[0079] Furthermore, in the embodiment of the present application, a possible implementation method of obtaining the two-dimensional point cloud of the target face in step S12 is exemplarily introduced as follows.

[0080] Based on the three-dimensional point cloud of the first part of the target face, a two-dimensional point cloud of the first part of the target face is obtained. Specifically, the projection of the three-dimensional point cloud of the first part in the two-dimensional face image can be used as the two-dimensional point cloud of the first part of the target face. A two-dimensional point cloud detector can be used to obtain a two-dimensional point cloud of the second part of the target face, and the two-dimensional point cloud of the first part and the two-dimensional point cloud of the second part form the two-dimensional point cloud of the target face.

[0081] Further, in the present embodiment, the following is constructed: Figure 3 The mapping relationship between the three-dimensional point cloud and the two-dimensional point cloud is shown, and the final three-dimensional point cloud is obtained by solving the scaling parameter, rotation matrix, offset matrix, shape coefficient and expression coefficient.

[0082] For details, please refer to Figure 4 , Figure 4 The sub-step flow chart of step S13 is illustrated. In the embodiment of the present application, step S13 can be implemented through the following sub-steps.

[0083] Sub-step S131, calculating the error value between the projection of the three-dimensional point cloud corresponding to the target face on the two-dimensional plane and the two-dimensional point cloud.

[0084] Among them, the projection L of the three-dimensional point cloud on the two-dimensional plane 1 The calculation formula can be as follows:

[0085]

[0086] The error value between the projection of the three-dimensional point cloud on the two-dimensional plane and the two-dimensional point cloud can be expressed as the norm of the difference between the projection of the three-dimensional point cloud on the two-dimensional plane and the two-dimensional point cloud.

[0087] Specifically, the error value can be expressed as ||L1 -L 2 ||, where L 2 It is a two-dimensional point cloud.

[0088] Sub-step S132, optimizing the error value by adjusting the three-dimensional face prediction parameters of the target face so that the error value meets the preset conditions.

[0089] In this sub-step, the error value is optimized by adjusting the expression coefficient of the target face and keeping other parameters of the three-dimensional face prediction parameters of the target face except the expression coefficient unchanged, so that the error value can be minimized.

[0090]

[0091] Among them, the scaling parameter s, the rotation matrix R, the offset matrix t 2d and shape factor s i unchanged, we can just adjust the expression coefficient e i Make the error value reach the minimum value.

[0092] In this sub-step, specific optimization methods include gradient descent method, least squares method, non-negative least squares method and L-BFGS algorithm.

[0093] Sub-step S133, performing face reconstruction based on the adjusted three-dimensional face prediction parameters of the target face to obtain a corrected three-dimensional point cloud of the target face.

[0094] The above method uses the points in the three-dimensional point cloud to construct the mapping relationship between the three-dimensional point cloud and the two-dimensional point cloud through the three-dimensional point cloud of the target face that has been constructed. The two-dimensional point cloud of the second part (for example, the mouth) is obtained through the two-dimensional key point detector, that is, only one two-dimensional point cloud input of the second part is needed to construct the mapping relationship between the three-dimensional point cloud and the two-dimensional point cloud of the entire target face. After the mapping relationship is constructed, the expression coefficient e can be solved i The reason why we only solve the expression coefficient e i Instead of solving the scaling parameter s, rotation matrix R, offset matrix t 2d and shape factor s i The reason is that there is no need to solve these parameters. These parameters can be directly used as the output of the 3D face prediction network model as known quantities. i , will take into account the first part (for example, the nose contour) and the second part. When adjusting, since the first part has not changed much, the second part is mainly adjusted. Adding the first part can play the following role: when the 2D point cloud of the second part is out of place (for example, the 2D point cloud is not in the face area), because of the constraint that the first part does not change, the expression coefficient e iThe coefficients obtained must also ensure the invariance of the first part. In this way, the purpose of adjusting the second part can be achieved on the basis of ensuring that the first part remains unchanged.

[0095] The above solution provided in the embodiment of the present application adopts the three-dimensional point cloud predicted by the three-dimensional face prediction network model based on deep learning to ensure that the predicted overall face posture and the actual face shape fit well, by only solving the expression coefficient e i Optimize the fit between the second part and the actual part, and improve the stability and accuracy of the second part, such as Figure 5 The schematic diagram of the effect of clown mask fitting based on the technical solution provided in the embodiment of the present application is shown. Since only one parameter needs to be solved, the entire solution process becomes relatively simple and time-consuming, which can well meet the needs of business scenarios (such as live broadcast scenarios) for low memory and low time consumption.

[0096] For further information, please refer to Figure 6 , Figure 6 A schematic diagram of a functional module of a three-dimensional face reconstruction device 200 provided in an embodiment of the present application. In an embodiment of the present application, the three-dimensional face reconstruction device 200 can be divided into functional modules according to the method embodiment executed by the electronic device, that is, the following functional modules corresponding to the three-dimensional face reconstruction device 200 can be used to execute the above-mentioned method embodiments. Among them, the three-dimensional face reconstruction device 200 can include a prediction module 210, an acquisition module 220 and a correction module 230. The functions of each functional module of the three-dimensional face reconstruction device 200 are described in detail below.

[0097] The prediction module 210 is used to input a frame of two-dimensional face image into a trained three-dimensional face prediction network model to predict facial key points to obtain a three-dimensional point cloud of the target face in the two-dimensional face image.

[0098] Among them, the two-dimensional face image can be sampled from the video image, for example, it can be sampled from the live video of the anchor, and the sampling frequency can be set according to actual needs, for example, one frame of two-dimensional face image can be sampled every 5 frames.

[0099] The three-dimensional face prediction network model can be trained based on the deep learning network model. The overall facial posture represented by the three-dimensional point cloud of the target face fits well with the actual face shape. The three-dimensional point cloud includes a large number of three-dimensional key points of the face, and each three-dimensional key point of the face includes three-dimensional coordinates. The facial contour represented by the three-dimensional point cloud is the contour of the real face, which will be hidden behind the target face when the angle is large.

[0100] The target face may include a first part that does not need to be corrected and a second part to be corrected, wherein the second part to be corrected refers to a part with a larger range of movement, such as the mouth and eyes; the first part is the other part of the target face excluding the second part, such as the nose and facial contour.

[0101] In this embodiment, the prediction module 210 may be used to execute the above step S11. The detailed implementation of the prediction module 210 may refer to the above detailed description of step S11.

[0102] The acquisition module 220 is used to acquire the two-dimensional point cloud of the target face.

[0103] The two-dimensional point cloud includes a large number of two-dimensional key points of the face, and each two-dimensional key point of the face includes a two-dimensional coordinate corresponding to the two-dimensional face image. The two-dimensional point cloud of the first part is obtained in a different manner from the two-dimensional point cloud of the second part, and the facial contour represented by the two-dimensional point cloud is always located at the facial edge of the target face.

[0104] In this embodiment, the acquisition module 220 can be used to execute the above step S12. The detailed implementation of the acquisition module 220 can refer to the detailed description of the above step S12.

[0105] The correction module 230 is used to correct the three-dimensional point cloud of the second part of the target face based on the three-dimensional point cloud and the two-dimensional point cloud of the target face to obtain a corrected three-dimensional point cloud of the target face.

[0106] Among them, the correction module 230 constructs a mapping relationship between the three-dimensional point cloud of the target face and the corresponding two-dimensional point cloud, and corrects the three-dimensional point cloud of the target face predicted by the three-dimensional face prediction network model through the above mapping relationship to obtain the corrected three-dimensional point cloud of the target face.

[0107] In this embodiment, the correction module 230 can be used to execute the above step S13. The detailed implementation of the correction module 230 can refer to the detailed description of the above step S13.

[0108] It should be noted that it should be understood that the division of the various modules in the above device or system is only a division of logical functions. In actual implementation, they can be fully or partially integrated into one physical entity, or they can be physically separated. And these modules can all be implemented in the form of software (for example, open source software) that can be called by the processor; they can also be all implemented in the form of hardware; some modules can also be implemented in the form of software called by the processor, and some modules can be implemented in the form of hardware. As an example, the correction module 230 can be implemented by running a separate processor, and can be stored in the memory of the above device or system in the form of program code. The functions of the above correction module 230 are called and executed by a processor of the above device or system. The implementation of other modules is similar to it, which will not be repeated here. In addition, these modules can be fully or partially integrated together, or they can be implemented independently. The processor described here can be an integrated circuit with signal processing capabilities. In the implementation process, each step or each module in the above technical solution can be completed by an integrated logic circuit in the processor or by executing a software program.

[0109] Please refer to Figure 7 , Figure 7 FIG. 1 is a schematic diagram of the hardware structure of an electronic device 100 for implementing the above-mentioned three-dimensional face reconstruction method provided by an embodiment of the present disclosure. Figure 7 As shown, the electronic device 100 may include a processor 110 , a computer-readable storage medium 120 , and a bus 130 .

[0110] In a specific implementation process, the processor 110 executes the computer-executable instructions (eg, Figure 6 The three-dimensional face reconstruction device 200 shown in the figure includes a prediction module 210, an acquisition module 220 and a correction module 230), so that the processor 110 can execute the three-dimensional face reconstruction method of the above method embodiment, wherein the processor 110 and the computer-readable storage medium 120 can be connected through a bus 130.

[0111] The specific implementation process of the processor 110 can refer to the various method embodiments executed by the above-mentioned electronic device 100. The implementation principles and technical effects are similar, and the embodiments of the present application will not be repeated here.

[0112] The computer readable storage medium 120 may be, but is not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable read-only memory (EEPROM), etc. Among them, the memory 111 is used to store programs or data.

[0113] The bus 130 can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, the buses in the drawings of the present application are not limited to only one bus or one type of bus.

[0114] In addition, an embodiment of the present application further provides a readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the three-dimensional face reconstruction method as described above is implemented.

[0115] In summary, the three-dimensional face reconstruction method, electronic device, and computer-readable storage medium provided in the embodiments of the present application divide the target face in the two-dimensional face image into a first part that does not need to be corrected and a second part to be corrected. After predicting the three-dimensional point cloud of the target face, the three-dimensional point cloud of the second part of the target face is corrected to obtain the corrected three-dimensional point cloud of the target face. In this way, the above method can correct the second part with a larger movement amplitude when the overall posture of the face predicted in the three-dimensional face prediction network model fits the actual face shape, optimize the fit effect between the predicted result of the part with a larger movement amplitude and the actual part, and take into account the fit effect between the predicted overall posture of the face and the local part.

[0116] The embodiments described above are only some embodiments of the present application, rather than all embodiments. The components of the embodiments of the present application generally described and shown in the drawings can be arranged and designed in various configurations. Therefore, the detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of protection of the present application, but merely represents the selected embodiments of the present application. Based on this, the scope of protection of the present application shall be based on the scope of protection of the claims. In addition, based on the embodiments of the present application, all other embodiments that can be obtained by those skilled in the art without making creative work should belong to the scope of protection of the present application.

Claims

1. A three-dimensional face reconstruction method, characterized in that: The method comprises: Input a frame of two-dimensional face image into a trained three-dimensional face prediction network model to predict face key points to obtain three-dimensional face prediction parameters of a target face in the two-dimensional face image, perform face reconstruction based on the three-dimensional face prediction parameters of the target face to obtain a three-dimensional point cloud projected onto the target face, wherein the target face includes a first part that does not need to be corrected and a second part that needs to be corrected, wherein the three-dimensional face prediction network model is trained based on a deep learning network model; Obtaining a two-dimensional point cloud of a first part of the target face based on the three-dimensional point cloud of the first part of the target face; Using a two-dimensional point cloud detector to obtain a two-dimensional point cloud of a second part of the target face, and obtaining a two-dimensional point cloud of the target face from the two-dimensional point cloud of the first part and the two-dimensional point cloud of the second part; Based on the mapping relationship between the three-dimensional point cloud and the two-dimensional point cloud of the target face, the three-dimensional point cloud of the second part of the target face is corrected to obtain a corrected three-dimensional point cloud of the target face.

2. The three-dimensional face reconstruction method according to claim 1, characterized in that: Before the step of inputting a frame of two-dimensional face image into the trained three-dimensional face prediction network model to predict face key points to obtain three-dimensional face prediction parameters of a target face in the two-dimensional face image, and performing face reconstruction based on the three-dimensional face prediction parameters of the target face to obtain a three-dimensional point cloud projected onto the target face, the method further includes the step of training the three-dimensional face prediction network model, which step includes: Acquire a training data set, the training data set comprising a plurality of two-dimensional face image samples carrying label parameters, the label parameters comprising face posture label parameters and face deformation label parameters; Inputting the two-dimensional face image samples into the deep learning network model to be trained for processing, and outputting three-dimensional face prediction parameters, wherein the three-dimensional face prediction parameters include face posture prediction parameters and face deformation prediction parameters; Performing face reconstruction based on the three-dimensional face prediction parameters to obtain a first three-dimensional point cloud projected onto the face in the two-dimensional face image sample, and performing face reconstruction based on the label parameters of the two-dimensional face image sample to obtain a second three-dimensional point cloud projected onto the face in the two-dimensional face image sample; The model loss function value of the deep learning network model is calculated by the first three-dimensional point cloud and the second three-dimensional point cloud, and when the model loss function value is greater than or equal to a preset loss function threshold, the model parameters of the deep learning network model are adjusted by back propagation of gradients, and the step of inputting the two-dimensional face image samples into the deep learning network model to be trained for processing in sequence and outputting three-dimensional face prediction parameters is returned to, until the model loss function value of the first three-dimensional point cloud and the second three-dimensional point cloud is less than the preset loss function threshold, and the trained deep learning network model is obtained as the three-dimensional face prediction network model.

3. The three-dimensional face reconstruction method according to claim 2, characterized in that: The step of correcting the three-dimensional point cloud of the second part of the target face based on the mapping relationship between the three-dimensional point cloud and the two-dimensional point cloud to obtain the corrected three-dimensional point cloud of the target face comprises: Calculating an error value between a projection of a three-dimensional point cloud corresponding to the target face on a two-dimensional plane and the two-dimensional point cloud; Optimizing the error value by adjusting the three-dimensional face prediction parameters of the target face so that the error value meets a preset condition; Face reconstruction is performed based on the adjusted three-dimensional face prediction parameters of the target face to obtain a corrected three-dimensional point cloud of the target face.

4. The three-dimensional face reconstruction method according to claim 3, characterized in that: The face deformation prediction parameter includes an expression coefficient, and the step of optimizing the error value by adjusting the three-dimensional face prediction parameter of the target face so that the error value satisfies a preset condition includes: The error value is optimized by adjusting the expression coefficient of the target face and keeping other parameters in the three-dimensional face prediction parameters of the target face except the expression coefficient unchanged, so that the error value reaches a minimum value, wherein the method for optimizing the error value includes gradient descent method, least squares method, non-negative least squares method and L-BFGS algorithm.

5. The three-dimensional face reconstruction method according to claim 4, characterized in that: The step of performing face reconstruction based on the adjusted three-dimensional face prediction parameters of the target face to obtain a corrected three-dimensional point cloud of the target face comprises: The expression coefficient corresponding to the time when the error value reaches the minimum value and other parameters of the three-dimensional face prediction parameters of the target face except the expression coefficient form the adjusted three-dimensional face prediction parameters of the target face; The adjusted three-dimensional face prediction parameters of the target face are used to perform face reconstruction to obtain a corrected three-dimensional point cloud of the target face.

6. The three-dimensional face reconstruction method according to claim 5, characterized in that: The face posture prediction parameters include scaling parameters, rotation matrix and offset matrix, and the face deformation prediction parameters also include shape coefficients. The formula used for face reconstruction is as follows: Among them, X is the projected 3D point cloud, s is the scaling parameter, P is the orthogonal projection matrix, R is the rotation matrix, S is the average face modeled by the PCA 3D model, which is a known quantity, and α i is the PCA basis of shape coefficients, s i is the shape coefficient, m is the dimension of the shape coefficient, β i is the PCA basis of the expression coefficient in the deformation parameter, e i is the expression coefficient, n is the dimension of the expression coefficient, t 2d is the offset matrix.

7. An electronic device, characterized in that: The electronic device includes a processor and a computer-readable storage medium, the processor and the computer-readable storage medium are connected via a bus system, the computer-readable storage medium is used to store programs, instructions or codes, and the processor is used to execute the programs, instructions or codes in the computer-readable storage medium to implement the three-dimensional face reconstruction method described in any one of claims 1-6.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions which, when executed, enable the server to execute the three-dimensional face reconstruction method described in any one of claims 1 to 6.

Images