3D face reconstruction and model training method, related equipment and program product

By using at least two autoencoders in 3D face reconstruction for 3D parameter regression and contrast learning constraints, the problem of insufficient stability of 3D face reconstruction in the prior art is solved, and a more efficient and stable reconstruction effect is achieved.

CN119048684APending Publication Date: 2024-11-29HEFEI IFLYTEK TOYCLOUD TECH
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202411248757.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-09-06
Publication Date
2024-11-29

AI Technical Summary

Technical Problem

The existing 3D face reconstruction methods have insufficient stability, especially the differentiable rendering method based on deep neural networks can easily lead to jitter in the reconstruction result, and high-precision reconstruction requires intensive key point models and high-cost hardware support.

Method used

At least two autoencoders (first autoencoder and second autoencoder) are used to perform 3DMM parameter regression on the original face image, and the model stability is improved through reconstruction loss and contrast learning constraints. The specific steps include obtaining training data. By regressing parameters by the autoencoder, face rendering and loss constraints are performed, and the first autoencoder is optimized by using a comparison learning strategy to enhance its extraction ability.

Benefits of technology

The stability and accuracy of the 3D face reconstruction model is improved, the dependence on key point models and hardware is reduced, and the stability and efficiency of the reconstruction results are improved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119048684A_ABST
    Figure CN119048684A_ABST
Patent Text Reader

Abstract

The invention discloses a 3D face reconstruction and model training method, a related device and a program product, at least two auto-encoders are designed in a training stage, a first type of 3DMM parameters are regressed from an original face image through the first auto-encoder, and the remaining 3DMM parameters are regressed from the original face image through the second auto-encoder, on one hand, the first type of 3DMM parameters are regressed from the original face image through the second auto-encoder, and on the other hand, the second type of 3DMM parameters are regressed from the original face image through the second auto-encoder; face rendering is carried out based on the complete 3DMM parameters, a reconstructed face image is obtained, and reconstruction constraint is carried out on the model; and on the other hand, the contrast learning strategy is adopted to carry out contrast learning constraint on the first auto-encoder, the stability of the first type of 3DMM parameters extracted by the first auto-encoder is enhanced, and on this basis, the stability of the reconstructed face image rendered based on the 3DMM parameters is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the technical field of 3D face reconstruction, and more specifically, to a 3D face reconstruction and model training method, related equipment and program products. Background Art

[0002] 3D face reconstruction is an important technology in the field of computer vision and graphics. It can restore a 3D face model from a single image or video sequence. 3D face reconstruction is widely used in security, virtual reality, and film and television production. It can improve the accuracy of face recognition, and can also be used for facial expression capture in movies and virtual try-on scenes in the fashion industry.

[0003] Existing 3D face reconstruction solutions are mainly based on the 3DMM model, which is a variable three-dimensional model that decouples the core features of the face and represents it with a series of vectors. By manipulating the coefficients of these vectors, the shape, texture and other features of the 3D face can be changed; the 3DMM model can also be used to fit the face in the real image by solving the vector coefficients. This process is also called 3D face reconstruction.

[0004] Commonly used 3D face reconstruction solutions based on 3DMM models are mainly divided into two categories: key point fitting and differentiable rendering. The key point fitting solution generally requires the use of a key point model to detect facial key points, and then solves the 3DMM parameters by using least squares and other optimization methods to reconstruct the face. During the optimization process, the parameter solution results of the previous frame can be explicitly used as constraints for the next frame through conditional constraints to avoid large changes in the parameter solution results of the next frame, thereby making the reconstructed face more stable. However, using this type of method for high-precision reconstruction usually requires a very dense key point model, and this type of model usually needs to be combined with hardware, such as a depth camera, or a large amount of high-quality and high-precision key point data for training, resulting in high technical requirements and costs.

[0005] Another common method usually combines deep neural networks and differentiable rendering, using a single autoencoder to regress all 3DMM parameters from a single frame image, then performs face rendering and calculates photometric loss with the original image, thereby back-propagating and optimizing the entire process. This type of method has low reliance on key points and can complete face reconstruction using existing open source models, so it is more widely used than solution 1. However, this type of method is prone to jitter in the reconstruction results, which makes the reconstructed face unstable. Summary of the invention

[0006] In view of the above problems, this application is proposed to provide a 3D face reconstruction and model training method, related equipment and program products to improve the stability of the reconstructed face. The specific solution is as follows:

[0007] In a first aspect, a 3D face reconstruction model training method is provided, wherein the 3D face reconstruction model includes at least a first autoencoder and a second autoencoder, and the method includes:

[0008] Acquire a training data set, the set including original face images;

[0009] Regressing the original face image to obtain a first type of 3DMM parameters through the first autoencoder, and regressing the original face image to obtain the remaining 3DMM parameters through the second autoencoder, wherein the first type of 3DMM parameters and the remaining 3DMM parameters constitute a complete 3DMM parameter;

[0010] Perform face rendering based on complete 3DMM parameters to obtain a reconstructed face image, and perform reconstruction loss constraints on the 3D face reconstruction model based on the reconstructed face image and the original face image;

[0011] The first autoencoder is subjected to comparative learning constraints with the goal of reducing the difference between the first type of 3DMM parameters of original facial images with the same target facial attributes and increasing the difference between the first type of 3DMM parameters of original facial images with different target facial attributes, wherein the target facial attributes are facial attributes represented by the first type of 3DMM parameters;

[0012] A 3D face reconstruction model trained with the reconstruction loss constraint and the contrastive learning constraint is obtained.

[0013] In one possible design, in another implementation of the first aspect of the embodiment of the present application, the first type of 3DMM parameter is a shape coefficient, and the face attribute represented by the shape coefficient is a face identity;

[0014] The 3D face reconstruction model also includes a third autoencoder for regressing to obtain expression coefficients, and the second autoencoder is specifically used to regress the original face image to obtain the remaining 3DMM parameters except the shape coefficient and the expression coefficient; the method also includes:

[0015] Regressing the original face image to obtain an expression coefficient through the third autoencoder;

[0016] With the goal of reducing the difference between expression coefficients of original facial images with the same expression and increasing the difference between expression coefficients of original facial images with different expressions, the third autoencoder is subjected to contrastive learning constraints.

[0017] In a possible design, in another implementation of the first aspect of the embodiment of the present application, the process of performing contrastive learning constraints on the first autoencoder includes:

[0018] Sampling a first anchor sample, a first positive sample, and a first negative sample from the training data set to form a first triplet, wherein the first anchor sample has the same face identity as the first positive sample and is different from the face identity of the first negative sample;

[0019] Each sample in the first triplet is respectively sent to the first autoencoder, and the shape coefficient corresponding to each sample is regressed, and the first triplet loss is used as the comparative learning loss to update the parameters of the first autoencoder, and the first triplet loss is used to shorten the distance between the first anchor sample and the first positive sample, and to increase the distance between the first anchor sample and the first negative sample.

[0020] In a possible design, in another implementation of the first aspect of the embodiment of the present application, the process of performing contrastive learning constraints on the third autoencoder includes:

[0021] Sampling a second anchor sample, a second positive sample, and a second negative sample from the training data set to form a second triplet, wherein the second anchor sample has the same expression as the second positive sample and is different from the expression of the second negative sample;

[0022] Each sample in the second triplet is sent to the third autoencoder respectively, and the expression coefficient corresponding to each sample is regressed, and the second triplet loss is used as the comparative learning loss to update the parameters of the third autoencoder. The second triplet loss is used to shorten the distance between the second anchor sample and the second positive sample, and to increase the distance between the second anchor sample and the second negative sample.

[0023] In a possible design, in another implementation of the first aspect of the embodiments of the present application, the process of sampling a first anchor sample, a first positive sample, and a first negative sample from the training data set includes:

[0024] Sampling a first anchor sample and a first positive sample from each original face image from the same video in the training data set, wherein the same video contains only one person and the expression remains unchanged;

[0025] An original face image with a different face identity from the first anchor sample is sampled from the training data set as a first negative sample.

[0026] In a possible design, in another implementation of the first aspect of the embodiments of the present application, the process of sampling a first anchor sample, a first positive sample, and a first negative sample from the training data set includes:

[0027] Sampling original face images with different accessory occlusion conditions under the same face identity from the training data set as the first anchor sample and the first positive sample respectively;

[0028] An original face image with a different face identity from the first anchor sample but the same accessory occlusion condition is sampled from the training data set as a first negative sample.

[0029] In a possible design, in another implementation of the first aspect of the embodiments of the present application, the process of sampling a first anchor sample, a first positive sample, and a first negative sample from the training data set includes:

[0030] Sampling original face images with different expressions under the same face identity from the training data set as the first anchor sample and the first positive sample respectively;

[0031] An original face image with a different face identity but the same expression as the first anchor sample is sampled from the training data set as a first negative sample.

[0032] In a possible design, in another implementation of the first aspect of the embodiments of the present application, the process of sampling the second anchor sample, the second positive sample, and the second negative sample from the training data set includes:

[0033] Sampling a second anchor sample and a second positive sample from each original face image from the same video in the training data set, wherein the same video contains only one person and the expression remains unchanged;

[0034] An original face image with a different expression from the second anchor sample is sampled from the training data set as a second negative sample.

[0035] In a possible design, in another implementation of the first aspect of the embodiments of the present application, the process of sampling the second anchor sample, the second positive sample, and the second negative sample from the training data set includes:

[0036] Sampling original face images with the same expression but different accessory occlusion conditions from the training data set as second anchor samples and second positive samples respectively;

[0037] An original face image having a different expression from that of the second anchor sample but the same accessory occlusion condition is sampled from the training data set as a second negative sample.

[0038] In a possible design, in another implementation of the first aspect of the embodiments of the present application, the process of sampling the second anchor sample, the second positive sample, and the second negative sample from the training data set includes:

[0039] Sampling original face images with the same expression but different face identities from the training data set as second anchor samples and second positive samples respectively;

[0040] An original face image having a different expression from the second anchor sample but the same face identity as the second anchor sample is sampled from the training data set as a second negative sample.

[0041] In a possible design, in another implementation of the first aspect of the embodiment of the present application, after obtaining the trained 3D face reconstruction model, the method further includes:

[0042] By using the knowledge distillation method, the first autoencoder, the second autoencoder and the third autoencoder in the trained 3D face reconstruction model are merged into a single autoencoder to obtain a simplified 3D face reconstruction model.

[0043] In a possible design, in another implementation manner of the first aspect of the embodiments of the present application, the training data set also includes pseudo labels of facial key points in the original facial image, and the method further includes:

[0044] Projecting the 3D mesh vertex coordinates of the reconstructed face image into the original face image to obtain the projected key point coordinates;

[0045] Based on the projected key point coordinates and the face key point pseudo labels, key point loss constraints are performed on the 3D face reconstruction model.

[0046] In one possible design, in another implementation of the first aspect of the embodiment of the present application, the reconstruction loss constraint includes: any one or more of a photometric loss constraint and a face recognition perceptual loss constraint.

[0047] In a second aspect, a 3D face reconstruction method is provided, comprising:

[0048] Obtaining an original face image to be reconstructed;

[0049] The original face image to be reconstructed is sent to a 3D face reconstruction model to obtain a reconstructed face image output by the model, and the 3D face reconstruction model is obtained using the 3D face reconstruction model training method described in any one of the first aspects of the present application.

[0050] In a third aspect, a 3D face reconstruction model training device is provided, wherein the 3D face reconstruction model includes at least a first autoencoder and a second autoencoder, and the device includes:

[0051] A training data acquisition unit, used to acquire a training data set, wherein the set includes an original face image;

[0052] A regression calculation unit, configured to regress a first type of 3DMM parameters from the original face image through the first autoencoder, and regress the remaining 3DMM parameters from the original face image through the second autoencoder, wherein the first type of 3DMM parameters and the remaining 3DMM parameters constitute complete 3DMM parameters;

[0053] A face rendering unit, used for performing face rendering based on complete 3DMM parameters to obtain a reconstructed face image;

[0054] A reconstruction loss constraint unit, used for performing reconstruction loss constraint on the 3D face reconstruction model based on the reconstructed face image and the original face image;

[0055] A first contrastive learning constraint unit is used to perform contrastive learning constraints on the first autoencoder with the goal of reducing the difference between the first type of 3DMM parameters of original facial images with the same target facial attributes and expanding the difference between the first type of 3DMM parameters of original facial images with different target facial attributes, wherein the target facial attributes are facial attributes represented by the first type of 3DMM parameters;

[0056] The model confirmation unit is used to obtain a 3D face reconstruction model after being trained with the reconstruction loss constraint and the contrastive learning constraint.

[0057] In a fourth aspect, an electronic device is provided, comprising: a memory and a processor;

[0058] The memory is used to store programs;

[0059] The processor is used to execute the program to implement the various steps of the 3D face reconstruction model training method described in any one of the first aspects of the present application, or to implement the various steps of the 3D face reconstruction method described in the second aspect of the present application.

[0060] In a fifth aspect, a readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the various steps of the 3D face reconstruction model training method described in any one of the first aspects of the present application are implemented, or the various steps of the 3D face reconstruction method described in the second aspect of the present application are implemented.

[0061] In a sixth aspect, a computer program product is provided, comprising a computer program. When the computer program is executed by a processor, the computer program implements the various steps of the 3D face reconstruction model training method described in any one of the first aspects of the present application, or implements the various steps of the 3D face reconstruction method described in the second aspect of the present application.

[0062] By means of the above technical scheme, the present application provides a 3D face reconstruction scheme based on the 3DMM model. In the 3D face reconstruction model training stage, unlike the traditional scheme that regresses all 3DMM parameters through a single autoencoder, the present application designs at least two autoencoders, defined as a first autoencoder and a second autoencoder. The first autoencoder regresses the first type of 3DMM parameters from the original face image in the training data set. The first type of 3DMM parameters can be types such as shape coefficients and expression coefficients. The second autoencoder regresses the remaining 3DMM parameters from the original face image. On this basis, on the one hand, face rendering is performed based on the complete 3DMM parameters to obtain a reconstructed face image. And based on the reconstructed face image and the original face image, the 3D face reconstruction model is reconstructed with constraints; on the other hand, a contrastive learning strategy is used to perform contrastive learning constraints on the first autoencoder. Specifically, the goal may be to reduce the difference between the first type of 3DMM parameters of the original face images with the same target face attributes, and to expand the difference between the first type of 3DMM parameters of the original face images with different target face attributes. The target face attribute is the face attribute represented by the first type of 3DMM parameter. For example, the first type of 3DMM parameter is the shape coefficient, and the corresponding face attribute may be the face identity. For another example, the first type of 3DMM parameter is the expression coefficient, and the corresponding face attribute may be the expression. Obviously, the present application introduces additional contrastive learning constraints on the first autoencoder during the training phase, which enhances the stability of the first type of 3DMM parameters extracted by the first autoencoder. On this basis, the stability of the reconstructed face image rendered based on the 3DMM parameters is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present application. Also, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0064] Figure 1 A schematic diagram of a process of a 3D face reconstruction model training method provided in an embodiment of the present application;

[0065] Figure 2 A schematic diagram of a model training architecture provided in an embodiment of the present application;

[0066] Figure 3 Another schematic diagram of the process of training a 3D face reconstruction model provided in an embodiment of the present application;

[0067] Figure 4 A schematic diagram of a network merging process based on knowledge distillation provided in an embodiment of the present application;

[0068] Figure 5 A schematic diagram of a process of performing comparative learning constraints on a first autoencoder Shape-Net provided in an embodiment of the present application;

[0069] Figure 6 A schematic diagram of a process of performing contrastive learning constraints on a third autoencoder Exp-Net provided in an embodiment of the present application;

[0070] Figure 7 A schematic diagram of the structure of a 3D face reconstruction model training device provided in an embodiment of the present application;

[0071] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0072] Before introducing the present application, the terms involved in the present application are first introduced:

[0073] 3DMM model: The full name is 3D Morphable Face Model, which is a statistical model of face shape and appearance. In 3DMM, there are many parameters used to control various aspects of the model to achieve high-precision face modeling and deformation. Examples of 3DMM parameters include shape coefficient, expression coefficient, texture coefficient, camera matrix, spherical harmonic illumination model coefficient, etc.

[0074] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0075] Commonly used 3D face reconstruction solutions based on 3DMM models are mainly divided into two categories: key point fitting and differentiable rendering. The key point fitting solution generally requires the use of a key point model to detect facial key points, and then solves the 3DMM parameters by using least squares and other optimization methods to reconstruct the face. During the optimization process, we can use conditional constraints to explicitly use the solution results of the previous frame's image parameters as constraints for the next frame to avoid large changes in the parameter solution results of the next frame, thereby making the reconstructed face more stable. However, using this type of method for high-precision reconstruction usually requires a very dense key point model, and this type of model usually needs to be combined with hardware, such as a depth camera, or a large amount of high-quality and high-precision key point data for training, resulting in high technical requirements and costs.

[0076] Another common method usually combines deep neural networks and differentiable rendering, using a single autoencoder to regress all 3DMM model parameters for a single frame image, and then performs face rendering and calculates photometric loss with the original image, thereby back-propagating to optimize the entire process. Since this type of method usually uses photometric loss as the main loss, key point constraint loss may not be applied, or key point constraint loss may be used as an auxiliary loss to optimize the network. The entire optimization process does not involve the collection and acquisition of real labels, and network training can be performed in a weakly supervised form. Compared with the previous solution, the number of points and accuracy requirements for the key point network are relatively low, and face reconstruction can be completed using existing open source models. Therefore, it is more widely used than solution one.

[0077] The second method based on differentiable rendering mentioned above usually performs 3DMM parameter regression and back propagation optimization on a single frame image input. This type of solution is usually difficult to use the results of the previous frame during training. If you want to use historical information, you can usually only use multi-frame averaging in the post-processing process. However, due to the lack of constraints on the parameter regression range, the post-processing method will inevitably cause jitter in the reconstruction result. In other words, the second method based on differentiable rendering mentioned above has the problem of unstable reconstruction results.

[0078] To this end, the present application provides a 3D face reconstruction method and a 3D face reconstruction model training method to improve at least some of the problems existing in the aforementioned examples and enhance the stability of the reconstruction results of the 3D face reconstruction model.

[0079] This application is mainly based on the differentiable rendering optimization process of traditional single-frame images. It does not use post-processing methods, but uses contrastive learning methods. Therefore, historical frames or other people's information can be introduced for comparison during the optimization process, which is used as a constraint loss for network training to improve the stability of the reconstruction results.

[0080] Since the traditional differentiable rendering solution uses a single network as an autoencoder to regress all 3DMM parameters including shape, expression, head pose, etc. at one time, this network structure makes it difficult to learn contrastive constraints. Therefore, this application proposes to decompose the original single autoencoder into at least two autoencoders, so that the single autoencoder can be optimized and constrained in a targeted manner. On this basis, the autoencoders can be further merged in the final stage by using knowledge distillation to improve the computational efficiency and versatility of the model.

[0081] The following embodiments of the present application provide a 3D face reconstruction method and a 3D face reconstruction model training method, respectively. The provided 3D face reconstruction model training method can be applied to an electronic device. For example, the electronic device can be a server, a robot, a smart phone, a personal computer (PC), a laptop, a wireless electronic device in an industrial control, etc.

[0082] The method provided in the present application can be divided into a training phase and a reasoning phase, wherein the training phase is the phase of training a 3D face reconstruction model, and the reasoning phase is the phase of using the trained 3D face reconstruction model to reconstruct a 3D face on an original face image. The training phase and the reasoning phase can be deployed in the same device or in different devices. For example, the training phase can be deployed in the cloud or in a server, and the reasoning phase can be deployed in an intelligent terminal, such as a mobile phone, a tablet, an augmented reality (AR) / virtual reality (VR) device, an intelligent robot, a vehicle-mounted terminal, or a wearable device.

[0083] It is understandable that before using the technical solutions disclosed in the embodiments of this application, the type, scope of use, usage scenarios, etc. of the personal information involved in this application should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0084] For ease of understanding, this application introduces the processes of the training phase and the reasoning phase respectively.

[0085] 1. Training Phase

[0086] It should be noted that, generally, the training process of the model can be divided into multiple iterations or multiple executions. This application exemplarily takes one of the iterations or execution processes as an example for illustrative introduction. The steps mentioned below can be repeated and will not be repeated below.

[0087] The 3D face reconstruction model training method provided in the embodiment of the present application, during the training stage, the 3D face reconstruction model may include at least two autoencoders, which are defined as a first autoencoder and a second autoencoder, respectively. The two autoencoders are respectively used to regress different types of 3DMM parameters for the input original image, and the set of 3DMM parameters regressed by the at least two autoencoders can constitute a complete 3DMM parameter.

[0088] See also Figure 1 , a flow chart of a training method for a 3D face reconstruction model provided in the present application is as follows, the method comprising:

[0089] Step S100: Acquire a training data set, wherein the set includes original face images.

[0090] Specifically, in order to subsequently train the 3D face reconstruction model, a training data set is obtained in this step, which includes original face images.

[0091] In order to improve the robustness of the trained 3D face reconstruction model, original face images in various scenarios can be collected in this step. The original face images can be obtained through open source datasets or extracted from public videos. For example, the original face image can be captured from a video containing only a single person's face.

[0092] In addition, the present application can also obtain the original face image by video recording. Specifically, the acquisition device is fixed in the shooting scene, and the camera component of the acquisition device is used to collect video data to ensure that only one person appears in each video, and the face of the person is centered and clearly visible as much as possible throughout the whole process, so as to ensure the quality of the original face image extracted subsequently. In each video, the person being collected needs to maintain the same expression as much as possible, and can constantly change the head posture to cover as many combinations as possible.

[0093] Furthermore, in order to improve the model's ability to reconstruct different facial images, in this step, videos with multiple expressions can be collected for the same person, such as videos with different expressions of joy, anger, sadness, and happiness.

[0094] Furthermore, in order to improve the model's ability to reconstruct occluded faces, in this step, you can also collect videos of the same person with multiple accessories occluding them. Examples of accessory occlusions include whether the person is wearing a mask, sunglasses, and other common objects. Of course, you can also collect videos of the same person with different expressions and different accessories occluding them.

[0095] At the same time, this step can also record the identity information of the person being collected in each video, the corresponding expression category, and the occlusion of accessories.

[0096] After obtaining the above video data, the video data can be frame extracted, deduplicated, and face detection can be performed. Optionally, the video frames can be further subjected to face key point detection. Here, there is no restriction on the algorithms used for face detection and face key point detection. In an optional implementation, the present application may use a pre-trained RetinaNet face detection algorithm and a FAN (FaceAlignment Network) face key point detection algorithm to obtain a face detection frame and corresponding face key point pseudo-labels on the captured video frames. These two algorithms can achieve an excellent balance in both accuracy and speed of face detection and face key point detection.

[0097] Furthermore, the face region is cropped according to the face detection result to obtain the cropped original face image. In this way, a training data set containing the original face image can be obtained. Optionally, for the face key point pseudo labels obtained in the video frame, they can be converted from the video frame to the cropped original face image to obtain the converted face key point pseudo labels, and the face key point expressions can also be added to the training data set and paired with the original face image to serve as training data together.

[0098] Further optionally, according to the facial identity, expression, and accessory occlusion conditions marked when the video data was shot, the cropped original facial image and the pseudo labels of facial key points can be divided into multiple subsets to ensure that the facial identity, expression, and accessory occlusion conditions of the original facial image in each subset are consistent, so as to facilitate use in the subsequent training stage.

[0099] Step S110: regress the first type of 3DMM parameters from the original face image through the first autoencoder, and regress the remaining 3DMM parameters from the original face image through the second autoencoder.

[0100] The first type of 3DMM parameters and the remaining 3DMM parameters constitute complete 3DMM parameters.

[0101] In this step, the first type of 3DMM parameters may be any type of parameters in the 3DMM parameters. Optionally, parameters that have a greater impact on the stability of face reconstruction may be selected from the 3DMM parameters. For example, the first type of 3DMM parameters may be shape coefficients, expression coefficients, etc. Among them, the shape coefficient affects the entire shape of the face, that is, it directly affects the identity of the reconstructed face, and the expression coefficient affects the expression type of the face. Both play an important role in the stability of the face reconstruction result.

[0102] Compared with the traditional solution of obtaining all 3DMM parameters through one autoencoder regression at one time, the regression calculation process of the first type of 3DMM parameters is separated in this application, and the regression calculation is performed through an independent first autoencoder, and the remaining 3DMM parameters are regressed using a second autoencoder. This facilitates the comparison learning constraints of the first autoencoder in the subsequent steps.

[0103] Step S120: performing face rendering based on the complete 3DMM parameters to obtain a reconstructed face image, and performing reconstruction loss constraints on the 3D face reconstruction model based on the reconstructed face image and the original face image.

[0104] Specifically, differentiable face rendering is performed based on the complete 3DMM parameters obtained in the previous step to obtain a reconstructed face image, wherein the complete 3DMM parameters may include: shape coefficient α, expression coefficient β, texture coefficient δ, camera matrix P, and spherical harmonic illumination model coefficient γ.

[0105] For reconstructing a face image, the 3D mesh vertex coordinates and mesh vertex texture can be expressed as the basic 3D mesh vertex coordinates and mesh vertex texture. In this application, the 3D mesh vertex coordinates can be expressed as B shape and B exp are shape basis vectors and expression basis vectors respectively, is the vertex coordinate of the average face model of BFM2009. The base texture of the 3D mesh vertex can be expressed as B t Represents the texture basis vector in BFM2009, represents the average texture of all vertices of the face model, and δ represents the texture coefficient. In addition to the basic texture of the vertex, the influence of lighting conditions also needs to be considered. For the 3D mesh vertex s i , normal n i and i The base texture at t i , the vertex s can be obtained by the following formula i The final texture at where γ b and Φ b Represents the coefficients and basis vector functions of the spherical harmonic illumination model. After obtaining the 3D mesh vertex coordinates, vertex final texture, and camera matrix P, the 3D face can be rendered using the differentiable rendering process, and the entire rendering process is differentiable.

[0106] For the basic optimization of the 3D face reconstruction model, the reconstruction loss can be used for constraint in this step. That is, the reconstruction loss constraint is applied to the 3D face reconstruction model based on the reconstructed face image and the original face image, and the network parameters of each module in the 3D face reconstruction model can be updated in this process.

[0107] The reconstruction loss constraint may include any one or more of a photometric loss constraint and a face recognition perceptual loss constraint.

[0108] For photometric loss, it can be used to measure the pixel difference between the reconstructed face image obtained by differentiable rendering and the original face image. The formula is expressed as:

[0109]

[0110] Wherein, M is the number of image pixels, I is the original face image, and I' is the reconstructed face image.

[0111] For face recognition perceptual loss, it can be used to measure the cosine similarity between the reconstructed face image and the original face image recognition representation. The formula is expressed as:

[0112]

[0113] Among them, I is the original face image, I' is the reconstructed face image, f represents the face recognition feature extraction network, and this application does not strictly limit the face recognition feature extraction network used here. Optionally, various types of face recognition feature extraction networks such as ArcFace can be used.

[0114] Step S130: Based on the contrastive learning strategy, perform contrastive learning constraints on the first autoencoder.

[0115] Specifically, based on the above-mentioned basic constraint training of the 3D face reconstruction model using reconstruction loss, a contrastive learning strategy is additionally introduced in this step to further perform contrastive learning constraints on the first autoencoder to improve the stability of the first type of 3DMM parameters output by the first autoencoder.

[0116] When the contrastive learning strategy is used to perform the contrastive learning constraint process on the first autoencoder, in order to avoid affecting other autoencoders, the other autoencoders can be frozen, such as freezing the parameters of the second autoencoder, and only updating the parameters of the first autoencoder.

[0117] The contrastive learning strategy can specifically be to perform contrastive learning constraints on the first autoencoder with the goal of reducing the difference between the first type of 3DMM parameters of original facial images with the same target facial attributes and expanding the difference between the first type of 3DMM parameters of original facial images with different target facial attributes.

[0118] Wherein, the target face attribute is the face attribute represented by the first type of 3DMM parameter. Taking the shape coefficient as the first type of 3DMM parameter as an example, the corresponding target face attribute can be the face shape, that is, the face identity (different face shapes correspond to different face identities). At this time, the first autoencoder is subjected to comparative learning constraints with the goal of reducing the difference between the shape coefficients of the original face images with the same face identity and expanding the difference between the shape coefficients of the original face images with different face identities.

[0119] For another example, taking the expression coefficient as the first type of 3DMM parameter, the corresponding target face attribute may be expression. At this time, the first autoencoder is subjected to comparative learning constraints with the goal of reducing the difference between the expression coefficients of original face images with the same expression and expanding the difference between the expression coefficients of original face images with different expressions.

[0120] Step S140: obtaining a 3D face reconstruction model after being trained with reconstruction loss constraints and contrastive learning constraints.

[0121] After training in the aforementioned steps according to the reconstruction loss constraints and the contrastive learning loss constraints, a trained 3D face reconstruction model is obtained.

[0122] The 3D face reconstruction model training method provided in the embodiment of the present application designs at least two autoencoders in the training stage, namely the first and second autoencoders. On the basis of training the model with the reconstruction loss constraint, a contrastive learning constraint on the first autoencoder is additionally introduced to enhance the stability of the first type of 3DMM parameters extracted by the first autoencoder. On this basis, the stability of the reconstructed face image rendered based on the 3DMM parameters is improved.

[0123] Optionally, the trained 3D face reconstruction model can be directly deployed in an application scenario to perform subsequent 3D face reconstruction tasks.

[0124] Another optional situation is that in order to improve the computational efficiency and versatility of the model, the present application can also adopt the knowledge distillation method to merge the respective encoders in the trained 3D face reconstruction model into a single autoencoder to obtain a simplified 3D face reconstruction model.

[0125] Specifically, the first autoencoder and the second autoencoder in the above-mentioned trained 3D face reconstruction model are frozen, and an untrained autoencoder network (such as a network structure such as R-Net) is distilled using knowledge distillation loss, so that the network can simultaneously predict the outputs of the first autoencoder and the second autoencoder. This case has no restrictions on the specific distillation algorithm, and without loss of generality, this case uses cross entropy loss as the knowledge distillation loss.

[0126] In some embodiments of the present application, if the training data set obtained in the aforementioned step S100 further includes pseudo labels of key points of the face corresponding to the original face image, then in step S120, based on the basic optimization of the 3D face reconstruction model using the reconstruction loss constraint, the present embodiment can further use the key point loss constraint to perform basic optimization on the 3D face reconstruction model. That is, in the basic optimization training process, the loss function can include both the reconstruction loss and the key point loss.

[0127] Specifically, in step S120, face rendering is performed based on the complete 3DMM parameters to obtain a reconstructed face image. The 3D mesh vertex coordinates of the reconstructed face image can be projected into the original face image to obtain the projected key point coordinates. Then, based on the projected key point coordinates and the face key point pseudo labels, the key point loss constraints can be performed on the 3D face reconstruction model.

[0128] Before training, a set of serial numbers can usually be predefined on the 3DMM model, and the 3D mesh vertex coordinates corresponding to these serial numbers are used to calculate the loss with the pseudo labels of facial key points after being projected onto the original face image.

[0129] The key point loss can be expressed as:

[0130]

[0131] Where N is the number of key points in the pseudo-label of face key points. Taking the aforementioned face key point detection algorithm using the FAN algorithm as an example, the FAN algorithm has 68 key points. i Represents the weight coefficient corresponding to the i-th key point, q i represents the pseudo label of facial key points, q i 'Indicates the coordinates of the projected key points.

[0132] After basic optimization training of the 3D face reconstruction model is performed using the above reconstruction loss constraints and key point loss constraints, the contrastive learning constraint training process in step S140 may be further performed.

[0133] In this embodiment, in the process of basic optimization training of the 3D face reconstruction model, in addition to adopting the reconstruction loss constraint, a key point loss constraint is further added, which can improve the basic performance of the 3D face reconstruction model after basic optimization.

[0134] In some embodiments of the present application, another training method for a 3D face reconstruction model is introduced. Figure 2 As shown, in this embodiment, the 3D face reconstruction model in the training stage may include three autoencoders, which are defined as the first autoencoder, the second autoencoder and the third autoencoder. The first autoencoder is used for regression calculation of the shape coefficient, the third autoencoder is used for regression calculation of the expression coefficient, and the second autoencoder is used for regression calculation of the remaining 3DMM parameters except the shape coefficient and the expression coefficient.

[0135] In some possible expressions, the first autoencoder can also be called a shape autoencoder, defined as Shape-Net, the third autoencoder can also be called an expression autoencoder, defined as Exp-Net, and the second autoencoder can also be called a camera illumination texture autoencoder, defined as P-Net.

[0136] Combination Figure 3 As shown, the 3D face reconstruction model training method in this embodiment may include the following steps:

[0137] Step S200: Obtain a training data set, where the set includes original face images.

[0138] Step S200 in this embodiment corresponds to step S100 in the aforementioned embodiment, and is described in detail above and will not be repeated here.

[0139] Step S210: regress the shape coefficient of the original face image through the first autoencoder, regress the expression coefficient of the original face image through the third autoencoder, and regress the remaining 3DMM parameters of the original face image through the second autoencoder.

[0140] Compared with the scheme of the aforementioned embodiment, three independent autoencoders are designed in this embodiment. The shape coefficient can be regressed through the first autoencoder, the expression coefficient can be regressed through the third autoencoder, and the remaining 3DMM parameters can be regressed through the second autoencoder. In this way, the two more important 3DMM parameters, the shape coefficient and the expression coefficient, can be regressed and calculated through independent autoencoders, which makes it easier to compare and learn the constraints of the first and third autoencoders in subsequent steps.

[0141] Step S220: Perform face rendering based on the complete 3DMM parameters to obtain a reconstructed face image, and perform reconstruction loss constraints on the 3D face reconstruction model based on the reconstructed face image and the original face image.

[0142] Step S220 in this embodiment corresponds to step S120 in the aforementioned embodiment, and is described in detail above and will not be repeated here.

[0143] Step S230: Based on the contrastive learning strategy, contrastive learning constraints are respectively performed on the first autoencoder and the third autoencoder.

[0144] Compared with the aforementioned embodiment, in this embodiment, comparative learning constraints are respectively performed on the first autoencoder and the third autoencoder, so as to improve the stability of the shape coefficient output by the first autoencoder and improve the stability of the expression coefficient output by the third autoencoder.

[0145] It should be noted that the process of performing contrastive learning constraints on the first autoencoder and the third autoencoder can be executed separately. In the process of performing contrastive learning constraint training on the first autoencoder, in order to avoid affecting other autoencoders, the second and third autoencoders can be frozen; in the process of performing contrastive learning constraint training on the third autoencoder, the first and second autoencoders can be frozen.

[0146] When performing contrastive learning constraints on the first autoencoder, the contrastive learning constraints on the first autoencoder can be specifically performed with the goal of reducing the difference between the shape coefficients of original facial images with the same facial identity and expanding the difference between the shape coefficients of original facial images with different facial identities.

[0147] When performing contrastive learning constraints on the third autoencoder, the contrastive learning constraints on the third autoencoder can be specifically performed with the goal of reducing the difference between the expression coefficients of original facial images with the same expression and expanding the difference between the expression coefficients of original facial images with different expressions.

[0148] Step S240: obtaining a 3D face reconstruction model after being trained with reconstruction loss constraints and contrastive learning constraints.

[0149] In this embodiment, the 3D face reconstruction model is designed as a structure of three autoencoders during the training phase, the first and third autoencoders are used for regression calculation of shape coefficients and expression coefficients, respectively, and the second autoencoder is used for regression calculation of the remaining 3DMM parameters. Based on this structure, on the basis of optimizing the reconstruction loss constraint of the model (key point loss constraint can also be added), the process of comparative learning constraint optimization can be further added for the first and third autoencoders, respectively, so as to improve the stability of the shape coefficient and expression coefficient in the obtained 3DMM parameters, and improve the stability of the reconstructed face image output by the model.

[0150] The 3D face reconstruction model trained by the above embodiment can be directly deployed in the application scenario to perform subsequent 3D face reconstruction tasks.

[0151] Another optional situation is that in order to improve the computational efficiency and versatility of the model, the present application can also adopt the knowledge distillation method to merge the respective encoders in the trained 3D face reconstruction model into a single autoencoder to obtain a simplified 3D face reconstruction model.

[0152] Combination Figure 4 As shown, the first autoencoder Shape-Net, the second autoencoder P-Net, and the third autoencoder Exp-Net in the above-mentioned trained 3D face reconstruction model are frozen, and an untrained autoencoder network (such as a network structure such as R-Net) is distilled using knowledge distillation loss, so that the network can simultaneously predict the outputs of the first autoencoder, the second autoencoder, and the third autoencoder, that is, it can simultaneously predict all 3DMM parameters. This case has no restrictions on the specific distillation algorithm, and without loss of generality, this case can use cross entropy loss as the knowledge distillation loss.

[0153] In some embodiments of the present application, the process of performing comparative learning constraints on the first autoencoder and the third autoencoder is described respectively.

[0154] The contrastive learning constraint process can adopt a variety of contrastive learning strategies. This embodiment introduces a contrastive learning strategy based on triplet loss.

[0155] In the contrastive learning strategy based on triplet loss, it is necessary to obtain triplets, including anchor samples, positive samples, and negative samples. Among them:

[0156] Anchor Sample

[0157] Definition: Anchor samples are the core samples in contrastive learning and are the main objects that we want the model to learn and understand. During the training process, anchor samples are used as reference points to compare with other samples.

[0158] Function: The main function of anchor samples is to provide a clear comparison benchmark for the model. By calculating the similarity or distance between anchor samples and other samples, the model can learn how to distinguish different data points.

[0159] Positive Sample

[0160] Definition: A positive sample is a sample that is similar or related to an anchor sample. In contrastive learning, a positive sample usually belongs to the same category or has similar features as the anchor sample.

[0161] Role: The role of positive samples is to help the model learn how to identify samples that are similar to anchor samples. By narrowing the distance between anchor samples and positive samples or improving the similarity score between them, the model can learn to capture and represent the key features in the data.

[0162] Negative Sample

[0163] Definition: Negative samples are samples that are dissimilar or unrelated to the anchor samples. In contrastive learning, negative samples usually belong to different categories or have different characteristics from the anchor samples.

[0164] Function: The main function of negative samples is to help the model learn how to distinguish different data points. By pushing the distance between anchor samples and negative samples or reducing the similarity score between them, the model can learn to cluster similar samples together and disperse dissimilar samples.

[0165] This embodiment first introduces the contrastive learning constraint process of the first autoencoder.

[0166] In this embodiment, the face identity of the anchor sample is the same as that of the positive sample, but different from that of the negative sample.

[0167] First, a first anchor sample, a first positive sample, and a first negative sample are sampled from the training data set to form a first triplet, wherein the first anchor sample has the same face identity as the first positive sample and is different from the face identity of the first negative sample.

[0168] This embodiment provides three different sampling modes for the first triplet. You can select any one of the three sampling modes, or you can select a combination of two or three of them.

[0169] The first sampling method:

[0170] The first positive sample and the first anchor sample can be sampled from the same video, and the first negative sample can be sampled from any video of other face identities, so that the first autoencoder can obtain a stable shape coefficient prediction result in the same video sequence.

[0171] Specifically, a first anchor sample and a first positive sample are sampled from each original face image from the same video in the training data set obtained in the aforementioned step S100, wherein the same video contains only one person and the expression remains unchanged. An original face image with a different face identity from the first anchor sample is sampled from the training data set as a first negative sample.

[0172] The second sampling method:

[0173] The first positive sample and the first anchor sample can be sampled from subsets with different expressions under the same face identity, and the first negative sample can be sampled from a subset of another face identity with the same expression as the first anchor sample. The purpose is to enable the first autoencoder to still predict the same shape coefficient for the same person under different expressions, thereby reducing the impact of expression changes on face reconstruction.

[0174] Specifically, original face images with different expressions under the same face identity are sampled from the training data set obtained in the aforementioned step S100 as the first anchor sample and the first positive sample, respectively. Original face images with different face identities but the same expressions as the first anchor sample are sampled from the training data set as the first negative sample.

[0175] The third sampling method:

[0176] The first positive sample and the first anchor sample can be sampled from subsets with different accessory occlusion conditions under the same face identity, and the first negative sample can be sampled from a subset of another face identity with the same accessory occlusion condition as the anchor sample. The purpose is to enable the first autoencoder to still predict the same shape coefficient for the same person under different accessory occlusion conditions, thereby reducing the impact of facial occlusion on face reconstruction.

[0177] Specifically, original face images with different accessory occlusion conditions under the same face identity are sampled from the training data set obtained in the aforementioned step S100 as the first anchor sample and the first positive sample, respectively. Original face images with different face identities from the first anchor sample but the same accessory occlusion conditions are sampled from the training data set as the first negative sample.

[0178] After obtaining the first triplet through any one or more of the above sampling methods, the triplet can be used to perform contrastive learning constraints on the first autoencoder.

[0179] Specifically, each sample in the first triplet is sent to the first autoencoder, and the shape coefficient corresponding to each sample is regressed, and the first triplet loss is used as the contrastive learning loss to update the parameters of the first autoencoder. The first triplet loss is used to shorten the distance between the first anchor sample and the first positive sample, and to increase the distance between the first anchor sample and the first negative sample, such as Figure 5 shown.

[0180] In the process of using the first triplet to perform contrastive learning constraints on the first autoencoder, the reconstruction loss constraints and key point loss constraints introduced in the above embodiments can still be used simultaneously for optimization training. During the process, the parameters of the second autoencoder and the third autoencoder are frozen, and only the parameters of the first autoencoder are updated.

[0181] The first triplet loss can be expressed as:

[0182] L triplet =max(cos(a,p)-cos(a,n)+margin,0)

[0183] Among them, a, p, and n represent the shape coefficients of the first anchor sample, the first positive sample, and the first negative sample respectively, cos() represents the cosine similarity function, and margin is a constant greater than 0.

[0184] By combining the above sampling method and contrastive learning loss constraint, the first autoencoder Shape-Net can stably predict the same shape coefficient for the same person under various conditions, which improves the stability of the shape coefficient prediction results.

[0185] Furthermore, the contrastive learning constraint process of the third autoencoder is introduced.

[0186] In this embodiment, the expression of the anchor sample is the same as that of the positive sample, but different from that of the negative sample.

[0187] First, a second anchor sample, a second positive sample, and a second negative sample are sampled from the training data set to form a second triplet, wherein the second anchor sample has the same expression as the second positive sample and is different from the expression of the second negative sample.

[0188] This embodiment provides three different sampling modes for the second triplet. You can select any one of the three sampling modes, or you can select a combination of two or three of them.

[0189] The first sampling method:

[0190] The second positive sample and the second anchor sample can be sampled from the same video, and the second negative sample can be sampled from any video of other expressions. The purpose is to enable the third autoencoder to obtain a stable expression coefficient prediction result for the same expression in the same video sequence.

[0191] Specifically, a second anchor sample and a second positive sample are sampled from each original face image from the same video in the training data set obtained in the aforementioned step S100, wherein the same video contains only one person and the expression remains unchanged. An original face image with a different expression from the second anchor sample is sampled from the training data set as a second negative sample.

[0192] The second sampling method:

[0193] The second positive sample and the second anchor sample can be sampled from subsets with different accessory occlusion conditions under the same expression, and the second negative sample can be sampled from another subset of expressions with the same accessory occlusion condition as the second anchor sample. The purpose is to enable the third autoencoder to still predict the same expression coefficient for the same expression under different accessory occlusion conditions, thereby reducing the impact of facial occlusion on face reconstruction.

[0194] Specifically, original face images with the same expression but different accessory occlusion conditions are sampled from the training data set obtained in the aforementioned step S100 as the second anchor sample and the second positive sample, respectively. Original face images with different expressions from the second anchor sample but the same accessory occlusion conditions are sampled from the training data set as the second negative sample.

[0195] The third sampling method:

[0196] The second positive sample and the second anchor sample can be sampled from subsets with different face identities under the same expression, and the second negative sample can be sampled from a subset of another expression with the same face identity as the second anchor sample. The purpose is to enable the third autoencoder to still predict the same expression coefficient for the same expression under different face identities, thereby reducing the influence of face shape on face reconstruction.

[0197] Specifically, original face images with different facial identities under the same expression are sampled from the training data set obtained in the aforementioned step S100 as the second anchor sample and the second positive sample, respectively. Original face images with different expressions but the same facial identity as the second anchor sample are sampled from the training data set as the second negative sample.

[0198] After the second triplet is obtained by any one or more of the above sampling methods, the triplet can be used to perform contrastive learning constraints on the third autoencoder.

[0199] Specifically, each sample in the second triplet is sent to the third autoencoder to regress the expression coefficient corresponding to each sample, and the second triplet loss is used as the contrastive learning loss to update the parameters of the third autoencoder. The second triplet loss is used to shorten the distance between the second anchor sample and the second positive sample, and to increase the distance between the second anchor sample and the second negative sample, such as Figure 6 shown.

[0200] In the process of using the second triplet to perform contrastive learning constraints on the third autoencoder, the reconstruction loss constraints and key point loss constraints introduced in the above embodiments can still be used simultaneously for optimization training. During the process, the parameters of the first autoencoder and the second autoencoder are frozen, and only the parameters of the third autoencoder are updated.

[0201] The expression of the second triplet loss is the same as that of the first triplet loss. It is only necessary to replace a, p, and n with the expression coefficients of the second anchor sample, the second positive sample, and the second negative sample, respectively.

[0202] By combining the above sampling method and contrastive learning loss constraint, the third autoencoder Exp-Net can stably predict the same expression coefficient for the same type of expression under various conditions, which improves the stability of the expression coefficient prediction results.

[0203] Combining the contrastive learning constraints on the first and third autoencoders introduced in the above embodiment, it can be seen that the 3D face reconstruction model trained by the contrastive learning constraints can extract stable shapes and expression coefficients for the same person with the same expression in consecutive video frames without large jitters. At the same time, the comparison of different accessory occlusion conditions can also make the model more robust to face occlusion.

[0204] From the 3D face reconstruction model training methods introduced in the above embodiments, it can be seen that the present application can perform comparative constraints on the shape coefficients and expression coefficients of the 3D face reconstruction process during the model training stage, and can introduce historical frames in the same video or image frames in other videos for comparative learning by combining a variety of comparative learning sampling methods, so that the model has the ability to stably model 3D faces. At the same time, since the sampling strategy can consider samples under different accessory occlusion conditions, the model can learn factors related to the face but not related to occlusion, greatly improving the reconstruction effect of occluded faces and being more robust to face occlusion.

[0205] On this basis, the multiple autoencoder networks caused by the introduction of contrastive learning constraints can be further merged into a complete autoencoder using knowledge distillation, thereby improving the computational efficiency and versatility of the model.

[0206] 2. Reasoning Stage

[0207] On the basis of the 3D face reconstruction model training method introduced in any of the above embodiments, a 3D face reconstruction task can be performed based on the trained 3D face reconstruction model.

[0208] In some embodiments of the present application, a 3D face reconstruction method is further provided, specifically:

[0209] First, obtain the original face image to be reconstructed.

[0210] Furthermore, the original face image to be reconstructed is sent to the trained 3D face reconstruction model to obtain the reconstructed face image output by the model, that is, the reconstructed 3D face.

[0211] Since the training method of the aforementioned embodiment can improve the stability of the reconstruction result of the trained 3D face reconstruction model, when it is applied to the 3D face reconstruction task, it can ensure that the reconstructed 3D face is more stable and of higher quality.

[0212] The 3D face reconstruction model training device provided in an embodiment of the present application is described below. The 3D face reconstruction model training device described below and the 3D face reconstruction model training method described above can be referenced to each other.

[0213] See also Figure 7 , Figure 7 A schematic diagram of the structure of a 3D face reconstruction model training device disclosed in an embodiment of the present application.

[0214] The 3D face reconstruction model includes at least a first autoencoder and a second autoencoder. Figure 7 As shown, the device may include:

[0215] A training data acquisition unit 11 is used to acquire a training data set, wherein the set includes an original face image;

[0216] A regression calculation unit 12 is used to regress the first type of 3DMM parameters from the original face image through the first autoencoder, and regress the remaining 3DMM parameters from the original face image through the second autoencoder, wherein the first type of 3DMM parameters and the remaining 3DMM parameters constitute complete 3DMM parameters;

[0217] A face rendering unit 13, used for performing face rendering based on complete 3DMM parameters to obtain a reconstructed face image;

[0218] A reconstruction loss constraint unit 14, configured to perform reconstruction loss constraint on the 3D face reconstruction model based on the reconstructed face image and the original face image;

[0219] A first contrastive learning constraint unit 15 is used to perform contrastive learning constraints on the first autoencoder with the goal of reducing the difference between the first type of 3DMM parameters of original facial images with the same target facial attributes and expanding the difference between the first type of 3DMM parameters of original facial images with different target facial attributes, wherein the target facial attributes are facial attributes represented by the first type of 3DMM parameters;

[0220] The model confirmation unit 16 is used to obtain a 3D face reconstruction model after being trained with the reconstruction loss constraint and the contrastive learning constraint.

[0221] In one possible implementation, the first type of 3DMM parameter is a shape coefficient, and the face attribute represented by the shape coefficient is the face identity. The 3D face reconstruction model also includes a third autoencoder for regressing the expression coefficient, and the second autoencoder is specifically used to regress the original face image to obtain the remaining 3DMM parameters except the shape coefficient and the expression coefficient, and the regression calculation unit is also used to regress the original face image to obtain the expression coefficient through the third autoencoder. On this basis, the device of the present application may also include:

[0222] The second contrast learning constraint unit is used to perform contrast learning constraints on the third autoencoder with the goal of reducing the difference between expression coefficients of original facial images with the same expression and expanding the difference between expression coefficients of original facial images with different expressions.

[0223] In a possible implementation, the process of the first contrastive learning constraint unit performing contrastive learning constraints on the first autoencoder includes:

[0224] Sampling a first anchor sample, a first positive sample, and a first negative sample from the training data set to form a first triplet, wherein the first anchor sample has the same face identity as the first positive sample and is different from the face identity of the first negative sample;

[0225] Each sample in the first triplet is respectively sent to the first autoencoder, and the shape coefficient corresponding to each sample is regressed, and the first triplet loss is used as the comparative learning loss to update the parameters of the first autoencoder, and the first triplet loss is used to shorten the distance between the first anchor sample and the first positive sample, and to increase the distance between the first anchor sample and the first negative sample.

[0226] In a possible implementation, the process of the second contrastive learning constraint unit performing contrastive learning constraints on the third autoencoder includes:

[0227] Sampling a second anchor sample, a second positive sample, and a second negative sample from the training data set to form a second triplet, wherein the second anchor sample has the same expression as the second positive sample and is different from the expression of the second negative sample;

[0228] Each sample in the second triplet is sent to the third autoencoder respectively, and the expression coefficient corresponding to each sample is regressed, and the second triplet loss is used as the comparative learning loss to update the parameters of the third autoencoder. The second triplet loss is used to shorten the distance between the second anchor sample and the second positive sample, and to increase the distance between the second anchor sample and the second negative sample.

[0229] In some embodiments of the present application, several optional sampling methods are provided for the first contrastive learning constraint unit to sample the first anchor sample, the first positive sample and the first negative sample from the training data set, which are as follows:

[0230] The first sampling method includes:

[0231] Sampling a first anchor sample and a first positive sample from each original face image from the same video in the training data set, wherein the same video contains only one person and the expression remains unchanged;

[0232] An original face image with a different face identity from the first anchor sample is sampled from the training data set as a first negative sample.

[0233] The second sampling method includes:

[0234] Sampling original face images with different accessory occlusion conditions under the same face identity from the training data set as the first anchor sample and the first positive sample respectively;

[0235] An original face image with a different face identity from the first anchor sample but the same accessory occlusion condition is sampled from the training data set as a first negative sample.

[0236] The third sampling method includes:

[0237] Sampling original face images with different expressions under the same face identity from the training data set as the first anchor sample and the first positive sample respectively;

[0238] An original face image with a different face identity but the same expression as the first anchor sample is sampled from the training data set as a first negative sample.

[0239] In some embodiments of the present application, several optional sampling methods are provided for the second contrastive learning constraint unit to sample the second anchor sample, the second positive sample, and the second negative sample from the training data set, which are respectively as follows:

[0240] The first sampling method includes:

[0241] Sampling a second anchor sample and a second positive sample from each original face image from the same video in the training data set, wherein the same video contains only one person and the expression remains unchanged;

[0242] An original face image with a different expression from the second anchor sample is sampled from the training data set as a second negative sample.

[0243] The second sampling method includes:

[0244] Sampling original face images with the same expression but different accessory occlusion conditions from the training data set as second anchor samples and second positive samples respectively;

[0245] An original face image having a different expression from that of the second anchor sample but the same accessory occlusion condition is sampled from the training data set as a second negative sample.

[0246] The third sampling method includes:

[0247] Sampling original face images with the same expression but different face identities from the training data set as second anchor samples and second positive samples respectively;

[0248] An original face image having a different expression from the second anchor sample but the same face identity as the second anchor sample is sampled from the training data set as a second negative sample.

[0249] In some possible implementations, the apparatus of the present application may further include:

[0250] The knowledge distillation unit is used to merge the first autoencoder, the second autoencoder and the third autoencoder in the trained 3D face reconstruction model into a single autoencoder through a knowledge distillation method after obtaining the trained 3D face reconstruction model, so as to obtain a simplified 3D face reconstruction model.

[0251] In some possible implementations, the training data set also includes pseudo labels of facial key points in the original facial image, and the apparatus of the present application may also include:

[0252] A key point constraint unit is used to project the 3D mesh vertex coordinates of the reconstructed face image into the original face image to obtain the projected key point coordinates; based on the projected key point coordinates and the face key point pseudo-labels, perform key point loss constraints on the 3D face reconstruction model.

[0253] In some possible implementations, the reconstruction loss constraint includes: any one or more of a photometric loss constraint and a face recognition perceptual loss constraint.

[0254] The present application also provides an electronic device in an embodiment. Figure 8 As shown, it shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiment of the present application. The electronic device in the embodiment of the present application may include but is not limited to fixed terminals such as servers, tablet computers, mobile phones, etc. Figure 8 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0255] like Figure 8 As shown, the electronic device may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603 to implement the 3D face reconstruction model training method of the aforementioned embodiment of the present application, or implement the 3D face reconstruction method of the aforementioned embodiment of the present application. When the electronic device is powered on, various programs and data required for the operation of the electronic device are also stored in RAM 603. The processing device 601, ROM 602, and RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0256] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a memory card, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although Figure 8 An electronic device having various devices is shown, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0257] An embodiment of the present application also provides a computer program product, including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements any one of the 3D face reconstruction model training methods provided in the embodiments of the present application, or implements the 3D face reconstruction method of the aforementioned embodiments of the present application.

[0258] A computer-readable storage medium is also provided in an embodiment of the present application. The storage medium carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any 3D face reconstruction model training method provided in the embodiment of the present application, or implement the 3D face reconstruction method of the aforementioned embodiment of the present application.

[0259] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0260] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0261] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0262] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a training device, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, training device, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

[0263] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can refer to each other.

Claims

1. A 3D face reconstruction model training method, characterized in that: The 3D face reconstruction model includes at least a first autoencoder and a second autoencoder, and the method includes: Acquire a training data set, the set including original face images; Regressing the original face image to obtain a first type of 3DMM parameters through the first autoencoder, and regressing the original face image to obtain the remaining 3DMM parameters through the second autoencoder, wherein the first type of 3DMM parameters and the remaining 3DMM parameters constitute a complete 3DMM parameter; Perform face rendering based on complete 3DMM parameters to obtain a reconstructed face image, and perform reconstruction loss constraints on the 3D face reconstruction model based on the reconstructed face image and the original face image; The first autoencoder is subjected to comparative learning constraints with the goal of reducing the difference between the first type of 3DMM parameters of original facial images with the same target facial attributes and increasing the difference between the first type of 3DMM parameters of original facial images with different target facial attributes, wherein the target facial attributes are facial attributes represented by the first type of 3DMM parameters; A 3D face reconstruction model trained with the reconstruction loss constraint and the contrastive learning constraint is obtained.

2. The method according to claim 1, characterized in that The first type of 3DMM parameter is a shape coefficient, and the face attribute represented by the shape coefficient is a face identity; The 3D face reconstruction model also includes a third autoencoder for regressing to obtain expression coefficients, and the second autoencoder is specifically used to regress the original face image to obtain the remaining 3DMM parameters except the shape coefficient and the expression coefficient; the method also includes: Regressing the original face image to obtain an expression coefficient through the third autoencoder; With the goal of reducing the difference between expression coefficients of original facial images with the same expression and increasing the difference between expression coefficients of original facial images with different expressions, the third autoencoder is subjected to contrastive learning constraints.

3. The method according to claim 2, characterized in that The process of performing contrastive learning constraints on the first autoencoder includes: Sampling a first anchor sample, a first positive sample, and a first negative sample from the training data set to form a first triplet, wherein the first anchor sample has the same face identity as the first positive sample and is different from the face identity of the first negative sample; Each sample in the first triplet is respectively sent to the first autoencoder, and the shape coefficient corresponding to each sample is regressed, and the first triplet loss is used as the comparative learning loss to update the parameters of the first autoencoder, and the first triplet loss is used to shorten the distance between the first anchor sample and the first positive sample, and to increase the distance between the first anchor sample and the first negative sample.

4. The method according to claim 2, characterized in that: The process of performing contrastive learning constraints on the third autoencoder includes: Sampling a second anchor sample, a second positive sample, and a second negative sample from the training data set to form a second triplet, wherein the second anchor sample has the same expression as the second positive sample and is different from the expression of the second negative sample; Each sample in the second triplet is sent to the third autoencoder respectively, and the expression coefficient corresponding to each sample is regressed, and the second triplet loss is used as the comparative learning loss to update the parameters of the third autoencoder. The second triplet loss is used to shorten the distance between the second anchor sample and the second positive sample, and to increase the distance between the second anchor sample and the second negative sample.

5. The method according to claim 3, characterized in that: The process of sampling a first anchor sample, a first positive sample, and a first negative sample from the training data set includes: Sampling a first anchor sample and a first positive sample from each original face image from the same video in the training data set, wherein the same video contains only one person and the expression remains unchanged; An original face image with a different face identity from the first anchor sample is sampled from the training data set as a first negative sample.

6. The method according to claim 3, characterized in that: The process of sampling a first anchor sample, a first positive sample, and a first negative sample from the training data set includes: Sampling original face images with different accessory occlusion conditions under the same face identity from the training data set as the first anchor sample and the first positive sample respectively; An original face image with a different face identity from the first anchor sample but the same accessory occlusion condition is sampled from the training data set as a first negative sample.

7. The method according to claim 3, characterized in that The process of sampling a first anchor sample, a first positive sample, and a first negative sample from the training data set includes: Sampling original face images with different expressions under the same face identity from the training data set as the first anchor sample and the first positive sample respectively; An original face image with a different face identity but the same expression as the first anchor sample is sampled from the training data set as a first negative sample.

8. The method according to claim 4, characterized in that The process of sampling a second anchor sample, a second positive sample, and a second negative sample from the training data set includes: Sampling a second anchor sample and a second positive sample from each original face image from the same video in the training data set, wherein the same video contains only one person and the expression remains unchanged; An original face image with a different expression from the second anchor sample is sampled from the training data set as a second negative sample.

9. The method according to claim 4, characterized in that The process of sampling a second anchor sample, a second positive sample, and a second negative sample from the training data set includes: Sampling original face images with the same expression but different accessory occlusion conditions from the training data set as second anchor samples and second positive samples respectively; An original face image having a different expression from that of the second anchor sample but the same accessory occlusion condition is sampled from the training data set as a second negative sample.

10. The method according to claim 4, characterized in that The process of sampling a second anchor sample, a second positive sample, and a second negative sample from the training data set includes: Sampling original face images with the same expression but different face identities from the training data set as second anchor samples and second positive samples respectively; An original face image having a different expression from the second anchor sample but the same face identity as the second anchor sample is sampled from the training data set as a second negative sample.

11. The method according to claim 2, characterized in that After obtaining the trained 3D face reconstruction model, it also includes: By using the knowledge distillation method, the first autoencoder, the second autoencoder and the third autoencoder in the trained 3D face reconstruction model are merged into a single autoencoder to obtain a simplified 3D face reconstruction model.

12. The method according to any one of claims 1 to 11, characterized in that: The training data set also includes pseudo labels of key facial points in the original facial image, and the method further includes: Projecting the 3D mesh vertex coordinates of the reconstructed face image into the original face image to obtain the projected key point coordinates; Based on the projected key point coordinates and the face key point pseudo labels, key point loss constraints are performed on the 3D face reconstruction model.

13. The method according to any one of claims 1 to 11, characterized in that: The reconstruction loss constraint includes: any one or more of a photometric loss constraint and a face recognition perception loss constraint.

14. A 3D face reconstruction method, characterized in that: include: Obtaining an original face image to be reconstructed; The original face image to be reconstructed is sent to a 3D face reconstruction model to obtain a reconstructed face image output by the model, and the 3D face reconstruction model is obtained by using the 3D face reconstruction model training method according to any one of claims 1 to 13.

15. A 3D face reconstruction model training device, characterized in that: The 3D face reconstruction model includes at least a first autoencoder and a second autoencoder, and the device includes: A training data acquisition unit, used to acquire a training data set, wherein the set includes an original face image; A regression calculation unit, configured to regress a first type of 3DMM parameters from the original face image through the first autoencoder, and regress the remaining 3DMM parameters from the original face image through the second autoencoder, wherein the first type of 3DMM parameters and the remaining 3DMM parameters constitute complete 3DMM parameters; A face rendering unit, used for performing face rendering based on complete 3DMM parameters to obtain a reconstructed face image; A reconstruction loss constraint unit, used for performing reconstruction loss constraint on the 3D face reconstruction model based on the reconstructed face image and the original face image; A first contrastive learning constraint unit is used to perform contrastive learning constraints on the first autoencoder with the goal of reducing the difference between the first type of 3DMM parameters of original facial images with the same target facial attributes and expanding the difference between the first type of 3DMM parameters of original facial images with different target facial attributes, wherein the target facial attributes are facial attributes represented by the first type of 3DMM parameters; The model confirmation unit is used to obtain a 3D face reconstruction model after being trained with the reconstruction loss constraint and the contrastive learning constraint.

16. An electronic device, characterized in that: include: Memory and processor; The memory is used to store programs; The processor is used to execute the program to implement the various steps of the 3D face reconstruction model training method according to any one of claims 1 to 13, or to implement the various steps of the 3D face reconstruction method according to claim 14.

17. A readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the computer program implements the steps of the 3D face reconstruction model training method according to any one of claims 1 to 13, or implements the steps of the 3D face reconstruction method according to claim 14.

18. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the computer program implements the steps of the 3D face reconstruction model training method as described in any one of claims 1 to 13, or implements the steps of the 3D face reconstruction method as described in claim 14.

Citation Information

Cited By

  • Single-image 3D face reconstruction method based on GAN network

    CN120852663A