Three-dimensional face reconstruction method, electronic device and readable storage medium

Abstract

This application belongs to the field of artificial intelligence technology and discloses a three-dimensional face reconstruction method, electronic device, and readable storage medium. The method includes: determining a single two-dimensional face image; based on the two-dimensional face image, calling a three-dimensional face reconstruction model to generate a three-dimensional face model; wherein the three-dimensional face reconstruction model includes a multi-attribute collaborative learning module, which is used to optimize initial geometric parameters to obtain target geometric parameters; during optimization, the coordinates of three-dimensional key points are mapped to the geometric parameter space as geometric priors through a multi-layer perceptron projection layer to generate corrections to the initial geometric parameters; the target geometric parameters are the superposition of the initial geometric parameters and the corrections, and the initial geometric parameters and the coordinates of the three-dimensional key points are determined by the two-dimensional face image. This method improves the accuracy of the reconstructed three-dimensional face model.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a three-dimensional face reconstruction method, an electronic device, and a readable storage medium. Background Art

[0002] In recent years, 3D face reconstruction technology has made significant progress in the field of computer vision. Many deep learning-based 3D face reconstruction methods have achieved the generation of high-quality 3D face models from a single input image by combining geometric modeling and image texture generation.

[0003] However, when generating 3D face models, current multi-attribute collaborative learning methods for 3D face reconstruction primarily employ cascaded separable learning. This method uses independent network branches to predict key points and geometric parameters, which are then post-processed and concatenated. However, its core drawback is that key point prediction errors propagate along the processing chain, causing geometric parameter optimization to fall into local suboptimal solutions. This results in global structural distortion in the 3D face model. Summary of the Invention

[0004] The purpose of the embodiments of the present application is to provide a three-dimensional face reconstruction method, an electronic device and a readable storage medium, aiming to solve the problem of how to improve the accuracy of the reconstructed three-dimensional face model.

[0005] In order to solve the above technical problems, this application is implemented as follows:

[0006] In a first aspect, an embodiment of the present application provides a method for 3D face reconstruction, the method comprising:

[0007] Determine a single two-dimensional face image;

[0008] Based on the two-dimensional face image, a three-dimensional face reconstruction model is called to generate a three-dimensional face model; wherein,

[0009] The three-dimensional face reconstruction model includes a multi-attribute collaborative learning module, which is used to optimize the initial geometric parameters to obtain target geometric parameters; during the optimization,

[0010] Through the multi-layer perceptron projection layer, the coordinates of the three-dimensional key points are mapped to the geometric parameter space as geometric priors to generate corrections to the initial geometric parameters; the target geometric parameters are the superposition of the initial geometric parameters and the corrections, and the initial geometric parameters and the coordinates of the three-dimensional key points are determined by the two-dimensional face image.

[0011] Optionally, the three-dimensional face reconstruction model further includes a semantic feature cascade module; wherein,

[0012] The semantic feature cascade module is used to encode the two-dimensional face image to obtain a fusion feature of global and local features; the initial geometric parameters and the coordinates of the three-dimensional key points are decoded from the fusion feature.

[0013] Optionally, the semantic feature cascade module includes a backbone network, and the backbone network includes a feature extraction unit;

[0014] The feature extraction unit includes two or more bottleneck layers, and some bottleneck layers include an attention mechanism based on convolution blocks.

[0015] Optionally, the static convolution kernel in the bottleneck layer is replaced by a parameterized kernel generator.

[0016] Optionally, in the process of training the three-dimensional face reconstruction model, for the multi-attribute collaborative learning module, the training key point parameters and the training geometric parameters form a bidirectional constraint through a differentiable mapping matrix; wherein, the bidirectional constraint includes a forward constraint and a reverse gradient, and the training key point parameters and the training geometric parameters are determined by the training images in the training image data set.

[0017] Optionally, the loss function corresponding to the reverse gradient is:

[0018]

[0019] in, is the face mesh generated by training geometric parameters, To train the geometric parameters, is the differentiable mapping matrix; k are the coordinates of the training key points.

[0020] Optionally, the reverse gradient refers to feeding back the calculated gradient to the multi-attribute collaborative learning module to drive the iterative optimization of the training geometric parameters on the Riemannian manifold; wherein the gradient is calculated by the following formula:

[0021]

[0022] in, Represents the loss function A vector of partial derivatives with respect to the model parameters.

[0023] Optionally, the loss function corresponding to the forward constraint is:

[0024]

[0025] in, is the geometric reconstruction loss, To train the geometric parameters, GA face mesh generated by the training geometric parameters; is the key point guidance loss, To train key point parameters; and To balance the weight.

[0026] In a second aspect, an embodiment of the present application provides an electronic device, which includes a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the method described in the first aspect.

[0027] In a third aspect, an embodiment of the present application provides a computer-readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0028] Compared with the related art, the embodiment of the present application determines a single two-dimensional face image, and based on the two-dimensional face image, calls a three-dimensional face reconstruction model to generate a three-dimensional face model, thereby avoiding the problem of global structural distortion of the three-dimensional face model. Among them, the three-dimensional face reconstruction model includes a multi-attribute collaborative learning module, which is used to optimize the initial geometric parameters to obtain target geometric parameters; during the optimization, the coordinates of the three-dimensional key points are mapped to the geometric parameter space as a geometric prior through the multi-layer perceptron projection layer to generate corrections to the initial geometric parameters; the target geometric parameters are the superposition of the initial geometric parameters and the corrections, and the initial geometric parameters and the coordinates of the three-dimensional key points are determined by the two-dimensional face image. It can be understood that the present application corrects the geometric parameters through the coordinates of the three-dimensional key points, rather than the post-processing and splicing in the related art, that is, introducing a residual correction mechanism, and treating the coordinates of the three-dimensional key points as high-order correction terms rather than basic driving terms. This allows the prediction error of the three-dimensional key points to be compressed to a low-dimensional submanifold in the geometric parameter space, that is, the linear amplification of the error is suppressed through the contraction mapping principle, thereby blocking the cascade error transmission path from the source, helping to solve the problem of global structural distortion caused by local prediction deviations, thereby improving the accuracy of the reconstructed three-dimensional face model. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 A schematic diagram of a process flow of a three-dimensional face reconstruction method provided in an embodiment of the present application;

[0030] Figure 2 A schematic diagram of the structure of the progressive cascade of semantic features provided by the three-dimensional face reconstruction method provided in an embodiment of the present application;

[0031] Figure 3A schematic diagram of the structure of the attention mechanism based on convolutional blocks provided for the 3D face reconstruction method provided in an embodiment of the present application;

[0032] Figure 4 A schematic diagram of the structure of a feature extraction unit using dynamic convolution kernel adaptation provided in the 3D face reconstruction method provided in an embodiment of the present application;

[0033] Figure 5 A schematic diagram of the structure of a multi-attribute collaborative learning mechanism based on bidirectional constraints provided for the 3D face reconstruction method provided in an embodiment of the present application;

[0034] Figure 6 A schematic diagram of a 3D face reconstruction model structure of a 3D face reconstruction method provided in another embodiment of the present application;

[0035] Figure 7 A schematic diagram of the structure of a three-dimensional face reconstruction system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0036] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0037] The terms "first," "second," and the like in the specification and claims of this application are used to distinguish similar objects and are not used to describe a particular order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate so that the embodiments of this application can be implemented in an order other than those illustrated or described herein. In addition, the term "and / or" in the specification and claims refers to at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0038] In order to illustrate the technical solution described in this application, specific embodiments are provided below.

[0039] In an exemplary embodiment, Figure 1 As shown, a 3D face reconstruction method is provided, which involves the application stage of the 3D face reconstruction model, including steps 102 to 104.

[0040] Step 102: Determine a single two-dimensional face image.

[0041] The two-dimensional face image refers to a two-dimensional image containing a complete face, including but not limited to an image input by a user, an image extracted from a video file, or an image captured in real time by a camera.

[0042] Step 104: Based on the two-dimensional face image, call a three-dimensional face reconstruction model to generate a three-dimensional face model; wherein, the three-dimensional face reconstruction model includes a multi-attribute collaborative learning module, and the multi-attribute collaborative learning module is used to optimize the initial geometric parameters to obtain target geometric parameters; during optimization, the coordinates of the three-dimensional key points are mapped to the geometric parameter space as geometric priors through the multi-layer perceptron projection layer to generate corrections to the initial geometric parameters; the target geometric parameters are the superposition of the initial geometric parameters and the corrections, and the initial geometric parameters and the coordinates of the three-dimensional key points are determined by the two-dimensional face image.

[0043] Among them, during optimization, the coordinates of the three-dimensional key points are first predicted through independent network branches k , and the coordinates k As a geometric prior, where Secondly, the three-dimensional key points (low-dimensional) are mapped to the geometric parameter space (high-dimensional) to generate the correction amount of the initial geometric parameters ,in, ; Finally, the correction amount Superimposed to initial geometry parameters , forming the final target geometric parameters , target geometric parameters It is calculated by the following formula (1):

[0044] (1)

[0045] in, is a lightweight Multilayer Perceptron (MLP) encoder, W T is a matrix W The transpose of the matrix W It is the differentiable mapping matrix after model training is completed.

[0046] Compared with the related art, the embodiment of the present application determines a single two-dimensional face image, and based on the two-dimensional face image, calls a three-dimensional face reconstruction model to generate a three-dimensional face model, thereby avoiding the problem of global structural distortion of the three-dimensional face model. Among them, the three-dimensional face reconstruction model includes a multi-attribute collaborative learning module, which is used to optimize the initial geometric parameters to obtain target geometric parameters; during the optimization, the coordinates of the three-dimensional key points are mapped to the geometric parameter space as a geometric prior through the multi-layer perceptron projection layer to generate corrections to the initial geometric parameters; the target geometric parameters are the superposition of the initial geometric parameters and the corrections, and the initial geometric parameters and the coordinates of the three-dimensional key points are determined by the two-dimensional face image. It can be understood that the present application corrects the geometric parameters through the coordinates of the three-dimensional key points, rather than the post-processing and splicing in the related art, that is, introducing a residual correction mechanism, and treating the coordinates of the three-dimensional key points as high-order correction terms rather than basic driving terms. This allows the prediction error of the three-dimensional key points to be compressed to a low-dimensional submanifold in the geometric parameter space, that is, the linear amplification of the error is suppressed through the contraction mapping principle, thereby blocking the cascade error transmission path from the source, helping to solve the problem of global structural distortion caused by local prediction deviations, thereby improving the accuracy of the reconstructed three-dimensional face model.

[0047] Exemplarily, the three-dimensional face reconstruction model also includes a semantic feature cascade module; wherein, the semantic feature cascade module is used to encode based on the two-dimensional face image to obtain a fusion feature of global and local features; the initial geometric parameters and the coordinates of the three-dimensional key points are decoded from the fusion feature.

[0048] It's important to note that one of the core tasks of 3D face reconstruction is to effectively extract and express the geometric features of the face. However, due to the high degree of geometric nonrigidity and complexity of the face, its geometric features must not only include global information (such as the head outline and facial features), but also accurately capture local details (such as the curvature of the nose and the texture of the eye corners).

[0049] When dealing with the fusion of global and local features, related technologies often adopt a single scale or direct cascade approach, which makes it difficult to fully express the multi-level correlation of semantic features, resulting in significant deviations between the geometric details and overall consistency of the reconstruction results.

[0050] To this end, this embodiment proposes a semantic feature cascade module. This module uses encoding—that is, layer-by-layer extraction and fusion of global and local features—to form a hierarchical representation structure of semantic features. The core of this semantic feature cascade module is the combination of global contextual feature extraction and local detail feature cascade, capturing the diverse geometric shape information of faces at different scales. It also enhances feature interpretability and reconstruction accuracy through semantic region partitioning.

[0051] This embodiment provides two feature cascade methods: one is direct semantic feature cascade, and the other is progressive semantic feature cascade. The following describes the two feature cascade methods in detail. The details are as follows:

[0052] (1) Direct cascading of semantic features

[0053] In this method, by simply aligning and splicing features at different levels, multi-scale features can be quickly integrated. The specific implementation process is as follows:

[0054] First, assume that the backbone network consists of four feature extraction units, and the output features are: , , , .

[0055] The four feature extraction units are the first feature extraction unit, the second feature extraction unit, the third feature extraction unit and the fourth feature extraction unit. The process of extracting features from a two-dimensional face image by the backbone network is as follows: the two-dimensional face image is first input into the first feature extraction unit, and the first feature extraction unit extracts features from the two-dimensional face image. , then output the To the second feature extraction unit, the second feature extraction unit from Extracted from , and so on, finally the output of the third feature extraction unit is As the input of the fourth feature extraction unit, the fourth feature extraction unit extracts Extracted from .

[0056] in, C 1~ C 4 represents the number of channels of features output by the fourth feature extraction unit, the third feature extraction unit, the fourth feature extraction unit, and the first feature extraction unit, respectively.

[0057] Secondly, channel compression and downsampling operations are performed on the features of each layer (one feature extraction unit is a layer) to align the features output by each layer to a uniform resolution. , wherein, this embodiment has a resolution of There is no restriction on the specific value of . It should be noted that when the features are subsequently cascaded in the channel dimension, the features used must be of uniform resolution. Feature alignment is achieved through the following formula (2):

[0058] (2)

[0059] in, i= 1,2,3,4; Represents a downsampling operation.

[0060] Finally, the aligned features are concatenated in the channel dimension to obtain the final fusion features , the process is realized by the following formula (3):

[0061] (3)

[0062] in, Represents a concatenation operation in the channel dimension.

[0063] (2) Progressive cascading of semantic features

[0064] In this method, by gradually fusing feature maps from different resolutions, a closer connection is established between multi-scale features to enhance the overall feature expression capability. Figure 2 The specific implementation process is as follows:

[0065] First, channel compression and downsampling operations are performed on each layer of features to align their features to a uniform resolution. Subsequently, the resolution is adjusted through downsampling operations to obtain the aligned features as shown in formula (4):

[0066] (4)

[0067] in, i= 1,2,3,4; represents a 1×1 convolution operation, Represents a downsampling operation.

[0068] Then, starting from the first layer of features, we gradually fuse each layer of features to obtain the final fusion features. (Right now Figure 2 in ) is shown in formula (5):

[0069] (5)

[0070] in, represents pixel-wise addition and / or other weighted fusion operations, is the channel dimension after feature fusion.

[0071] Furthermore, the semantic feature cascade module includes a backbone network, which includes a feature extraction unit; the feature extraction unit includes two or more bottleneck layers, and some bottleneck layers include an attention mechanism based on convolution blocks.

[0072] In this embodiment, the feature extraction unit is MobileNetV3, which is a lightweight network model proposed by the Google team and has achieved excellent performance in tasks such as image classification, object detection, and semantic segmentation.

[0073] Reference Figure 3 , Figure 3 This figure shows the architecture of the Convolutional Block Attention Module (CBAM). CBAM is a lightweight attention mechanism for convolutional neural networks. By combining channel-wise and spatial-attention modules, it dynamically enhances important information in feature maps and suppresses irrelevant information. This allows for adaptive feature optimization, enabling the model to selectively focus on more important areas of the face, such as the eyes and nose.

[0074] For example, to balance performance and computational overhead, CBAM is only added to the bottleneck layers in the middle and deep layers of MobileNetV3, while not to the shallow layers, keeping computational overhead low. This allows the network to better focus on channels and spatial locations related to facial geometry, such as key areas like the eyes, nose, and mouth. This helps improve the accuracy of restoring key facial features in 3D face reconstruction tasks, especially when expressions vary significantly.

[0075] In some embodiments, for the strong personalized characteristics of facial identity and expression, refer to Figure 4 , employing dynamic convolution kernel adaptation, replacing the original static convolution with a parametric kernel generator (PKG). This feature extraction unit takes global average pooling features as input and dynamically generates 3×3 convolution kernel weights via a lightweight fully connected layer. In the implementation, the initial weight matrix serves as a learnable parameter, while a multi-layer perceptron generates dynamic offsets, further improving the accuracy of the 3D face model.

[0076] In some embodiments, during the training phase of the 3D face reconstruction model:

[0077] First, the training image is input to the encoder (the semantic feature cascade module mentioned above) and the independent decoder to regress a one-dimensional set of facial attribute parameters. , that is, training geometric parameters, the attributes corresponding to these parameters control the expression, posture and shape of the face. The parameters corresponding to these attributes are spliced ​​together to obtain , the splicing is achieved by formula (6):

[0078] (6)

[0079] Secondly, in order to further improve the accuracy of 3D face geometry reconstruction, the face geometry is predicted by the above-mentioned semantic feature cascade module based on the 3D face semantic features. The 3D face semantic features are obtained by preprocessing the training images (2D face images used for training); the 3D key points (corresponding to areas such as the nose, eyes, and mouth) are found through the face geometry, and the 3D key points of each area are respectively parsed into the corresponding training key point parameters ;Finally, the key point parameters will be trained As guidance information in the multi-attribute collaborative learning process to optimize the training geometric parameters .

[0080] Specifically, this embodiment trains geometric parameters and training keypoint parameters The collaborative optimization of further enhances the network's modeling capabilities for global geometry and local features. The optimization objective is shown in formula (7):

[0081] (7)

[0082] in, To train the geometric parameters, G is the face mesh generated by the training geometric parameters, It is the geometric reconstruction loss, which is used to ensure that the reconstructed face mesh is more similar to the real face.

[0083] is the key point guidance loss, Keypoint parameters are trained to ensure that the 3D keypoints projected by the model through the multi-layer perceptron projection layer are aligned with the 3D keypoints in the real image, resulting in a highly accurate 3D face model with respect to local features. 3D keypoint parameters are also used to construct collaborative constraints for geometric parameter learning, thereby improving the geometric accuracy of 3D face reconstruction.

[0084] and To balance the weights, it is used to adjust the relative importance of geometric reconstruction and key point constraints.

[0085] In the collaborative optimization process, by reducing To improve the performance of the model.

[0086] Furthermore, 3D face reconstruction technology still faces several other challenges, such as insufficiently accurate modeling of global geometry, poor recovery of local details (such as 3D keypoints), and insufficient decoupling between attributes. These issues result in the generated 3D face models lacking detail when expressing complex facial deformations and potentially lacking robustness to variations in the input image.

[0087] To alleviate these problems, in one exemplary embodiment, reference is made to Figure 5 , proposed a Multi-Attribute Collaborative Learning (MACL) mechanism based on bidirectional constraints. This mechanism combines multiple attributes (such as shape, expression, and posture) with 3D key point information, using collaborative optimization to achieve joint modeling of global geometry and local features, further improving the accuracy and detail of 3D face reconstruction.

[0088] Specifically, current research on multi-attribute collaborative learning mainly adopts cascaded separation learning or parallel multi-task learning.

[0089] For details about cascade separation learning, please refer to the description above.

[0090] Among them, although parallel multi-task learning shares the features extracted by the backbone network and performs parallel prediction of attributes, it lacks an explicit constraint mechanism between the attributes, which leads to feature competition between high-dimensional geometric parameters and low-dimensional key points.

[0091] In essence, the common problem between cascaded separation learning and parallel multi-task learning is the lack of geometric consistency, that is, there is a deformation decoupling between the sparse geometric constraints represented by three-dimensional key points and the dense surface controlled by the parameters of the FLAME face model, which leads to the coexistence of local detail distortion and global structure offset in the reconstruction result (i.e., the three-dimensional face model).

[0092] To address the problem of lack of geometric consistency, this embodiment proposes a three-dimensional key point guided geometric parameter collaborative learning framework. Through the multi-layer perceptron projection layer, the coordinates of the three-dimensional key points are mapped to the geometric parameter space as geometric priors, and residual terms are generated to correct the base facial geometric parameters; and a differentiable mapping matrix is ​​constructed to associate the mesh vertices in the FLAME face model with the three-dimensional key point topology, and a strong constraint term is introduced in the loss function.

[0093] It should be noted that a bidirectional constraint is formed between the 3D keypoints and the geometric parameters via a differentiable mapping matrix. The loss term constructs a Riemannian manifold constraint between the keypoints and the mesh vertices. During backpropagation, the geometric parameters and the 3D keypoints are simultaneously corrected to maintain conjugation on the optimization trajectory. This avoids suboptimal results caused by gradient conflicts, enables better iterative optimization of the model, and ensures the coordinated convergence of multi-attribute learning. This effectively alleviates the problem of lack of geometric consistency.

[0094] Among them, during the training phase of the model, it is necessary to predefine the differentiable mapping matrix ; That is, based on the 5023 vertices in the FLAME face template, define the matrix Elements in For encoding i Vertex pair j The contribution weight of each training key point.

[0095] Specifically, to further improve the accuracy of the reconstructed 3D face model, the multi-attribute collaborative learning module uses a differentiable mapping matrix to form bidirectional constraints on the training keypoint parameters and training geometric parameters. These bidirectional constraints include a forward constraint and a reverse gradient, and the training keypoint parameters and training geometric parameters are determined using training images from the training image dataset. The determination of these training keypoint parameters and training geometric parameters can be found in the description of the semantic feature cascade module above.

[0096] The bidirectional constraint is the geometric consistency loss, a comprehensive constraint that ensures the geometry of the generated 3D face model remains consistent with the input image (the input 2D face image) both globally and locally. It integrates multiple losses for geometry, key features, and unsupervised consistency (a self-constraint loss) to comprehensively optimize the shape information of 3D face reconstruction.

[0097] The loss function of the geometric consistency loss is shown in formula (8):

[0098] (8)

[0099] in, represents the geometric reconstruction loss, represents the keypoint guidance loss, Represents the consistency loss, each is the weight coefficient of the corresponding loss term.

[0100] The loss function corresponding to the forward constraint is the above formula 7.

[0101] Among them, the loss function corresponding to the reverse gradient is shown in formula (9):

[0102] (9)

[0103] in, is the face mesh generated by training geometric parameters, To train the geometric parameters, is the differentiable mapping matrix; k are the coordinates of the training key points. That is, in formula 8 .

[0104] The reverse gradient refers to feeding back the calculated gradient to the multi-attribute collaborative learning module to drive the iterative optimization of the training geometric parameters on the Riemannian manifold; wherein the gradient is calculated by the following formula (10):

[0105] (10)

[0106] in, Represents the loss function A vector of partial derivatives with respect to the model parameters.

[0107] In summary, the geometric consistency loss corresponding to the bidirectional constraint is the superposition of the loss corresponding to the forward constraint and the loss corresponding to the reverse gradient.

[0108] In addition, in some embodiments, in order to achieve accurate reconstruction of the shape and texture of the three-dimensional face model, a comprehensive loss function is also designed during the model training stage. The loss function includes image loss, facial key point reprojection loss, perceptual loss, geometric consistency loss and regularization loss.

[0109] By minimizing the following loss function, the geometric accuracy, texture naturalness, and visual consistency of the reconstruction results can be effectively improved. The comprehensive loss function is shown in formula (11):

[0110] (11)

[0111] in, represents the image loss, represents the reprojection loss of facial key points, represents the perceived loss, represents the geometric consistency loss, represents the regularization loss, each They represent the hyperparameters of each loss, which are used to adjust the weight of each loss in the total loss.

[0112] The following is a detailed description of each loss:

[0113] (1) Image loss

[0114] Image loss aims to guide the training process of 3D face model reconstruction by directly comparing image-level features. These losses include per-pixel color loss and sparse 2D keypoint loss, which optimize the model by directly quantifying the difference between the reconstructed image and the real image. Per-pixel color loss measures the difference between the reconstructed image and the real image at the pixel level.

[0115] Image loss usually uses mean square error to calculate the color difference of each pixel. Its purpose is to make the reconstructed image more consistent with the real image by minimizing the RGB color difference of each pixel.

[0116] Assumptions I is a real image, I * To reconstruct the image, each pixel position of the image is ( x,y ), then the loss function of each pixel color loss is shown in formula (12):

[0117] (12)

[0118] in, Represents the Euclidean distance, which is used to measure the difference between key points. H and W are the height and width of the image, I ( x,y )and I * ( x,y ) are the positions of the pixels in the real image and the reconstructed image, respectively.

[0119] Sparse 2D keypoint loss is mainly used to ensure that the key points (such as eyes, nose, mouth, etc.) in the reconstructed image are consistent with the key points in the real image. This loss considers a few sparse key points rather than the entire image. The 2D keypoint positions are usually extracted by the face detector. The loss function of sparse 2D keypoint loss is shown in formula (13):

[0120] (13)

[0121] in, represents the Euclidean distance, and They are i The 2D keypoint coordinates in the real image and the reconstructed image.

[0122] The loss function of image loss constructed by combining these two losses is shown in formula (14):

[0123] (14)

[0124] in, is a hyperparameter used to adjust the weight of sparse 2D keypoint loss in the total loss.

[0125] (2) Face key point reprojection loss

[0126] The facial keypoint reprojection loss is a loss function commonly used in 3D face reconstruction. It ensures that the locations of the feature points (including keypoints) of the reconstructed 3D face are consistent with the true locations of the feature points (i.e., the locations of the feature points in the input image) when projected onto a 2D plane. By calculating the Euclidean distance between the keypoints of the reconstructed face and the keypoints detected in the input image, this loss effectively constrains the facial geometry.

[0127] Assuming there are n key points, the reconstructed three-dimensional feature points are , the two-dimensional feature point labels in the training image are , then the loss function of the face key point reprojection loss is shown in formula (15):

[0128] (15)

[0129] in, Represents the weight of each facial key point. The weight of points in the mouth and nose area is set to 20, while the weight of other points is set to 1; represents the Euclidean distance; The first i The position of a three-dimensional feature point projected onto a two-dimensional plane.

[0130] (3) Perceptual loss

[0131] Perceptual loss is a loss function used to optimize reconstruction results, which focuses on the visual effects at the human perception level rather than just pure geometric reconstruction.

[0132] Traditional loss functions (such as reconstruction loss, feature point loss, etc.) usually focus on point-to-point or pixel-to-pixel differences. Although this usually produces good results, in 3D face reconstruction based on convolutional neural networks, this approach may lead to local optimal solutions.

[0133] The perceptual loss focuses more on the naturalness and realism of the reconstruction results, making the reconstructed face as close to the real face as possible. By using a pre-trained model to extract facial features from the input image I and reconstruct facial features in the image I * , and then calculate the cosine distance between these feature vectors. The loss function of perceptual loss is shown in formula (16):

[0134] (16)

[0135] in, represents the pre-trained model, Represents the vector inner product.

[0136] (4) Geometric consistency loss

[0137] Please refer to the description of geometric consistency loss in the above embodiment, which will not be repeated here.

[0138] (5) Regularization loss

[0139] The regularization loss aims to prevent the shape and texture degradation of the 3D face model generated by the 3D face reconstruction model. Therefore, regularization constraints are imposed on the identity parameters, expression parameters, and texture parameters respectively to ensure that the generated result has a good geometric structure and natural texture. The loss function of the regularization loss is shown in formula (17):

[0140] (17)

[0141] in, 、 and Represent identity parameters , expression parameters and texture parameters The weight of .

[0142] In an exemplary embodiment, Figure 6 As shown, a three-dimensional face reconstruction method is provided.

[0143] Figure 6 The overall architecture of the 3D digital face geometry reconstruction network based on semantic feature cascade and multi-attribute collaborative learning is presented. It mainly includes two modules: the gradually fused semantic feature cascade strategy (Semantic Feature Cascade, SFC) and MACL. Figure 6 The encoder in the module is the SFC module.

[0144] Among them, 601 is the global feature, 602 is the local feature, ɑ is the albedo parameter, c are the camera parameters, ψ is the expression parameter, l is the lighting parameter, β is the shape parameter, θ is the posture parameter.

[0145] In the feature extraction stage, the preprocessing correlation network processes the input image, extracts global features and semantic features, and then passes the semantic features step by step through the SFC module to improve the global and local correlation.

[0146] During the parameter collaborative learning phase, the MACL module optimizes and updates the model parameters of the 3D face reconstruction model, guiding model learning through semantic features to improve the geometric consistency of the 3D face. During the training phase, the model first extracts global and local features from the input image, obtains semantic features through a step-by-step feature cascade, and uses these features to guide model learning. Then, image loss and perceptual loss are used to constrain and optimize the 3D face parameters (i.e., geometric parameters) at both the pixel level and high-level perceptual levels. Finally, based on the optimized parameters, the final 3D face image (i.e., 3D face model) is rendered using a differential renderer.

[0147] Specifically, many current deep learning-based reconstruction methods still face the following key issues in practical applications:

[0148] (1) Insufficient fusion of semantic features leads to limited geometric reconstruction accuracy.

[0149] Current 3D face reconstruction methods rely heavily on low-level visual features. While these methods perform well on standard reconstruction tasks, they often fail to accurately capture the geometric structure of faces with significant expression variations, angle changes, and individual differences. This is especially true when training data is relatively simple or lacks complex facial samples. The limitations of low-level features often lead to distortion or inaccuracy in facial geometry. Furthermore, existing semantic feature fusion methods mostly combine features from different levels through simple concatenation or concatenation, but this approach fails to effectively capture high-level semantic information, thus affecting the final reconstruction results.

[0150] (2) Insufficient collaborative modeling among multiple attributes.

[0151] 3D face reconstruction involves multiple factors, such as identity, expression, and pose. Existing methods typically decouple these attributes and treat them separately. This lack of effective modeling of the complex interactions between these attributes prevents them from fully embodying the benefits of multi-attribute collaboration on geometry generation. This lack of a multi-attribute collaborative learning mechanism results in poor generalization and adaptability to complex scenarios.

[0152] In response to the above challenges, this embodiment proposes a 3D face geometry reconstruction method based on semantic feature cascade and multi-attribute collaborative learning. First, through the method of global and local feature cascade, an SFC module is designed to fully utilize the synergy between features at different levels and improve the model's ability to express geometric details. Secondly, a MACL module is proposed to improve the accuracy of parameter estimation and the generalization ability of the model by integrating multiple attribute information such as identity, expression and posture. Finally, combined with differentiable rendering technology, a comprehensive loss function is jointly optimized to ensure that the generated results are consistent with the target image at both the pixel level and the perceptual level, thus achieving high-quality 3D face geometry reconstruction.

[0153] Finally, it should be emphasized that this embodiment addresses the problem that existing three-dimensional face reconstruction methods are difficult to fully explore the correlation between multi-level semantic features and multiple attributes during the process of face geometry reconstruction, resulting in inaccurate reconstructed face geometry. A face geometry reconstruction method based on semantic feature cascade and multi-attribute collaborative learning is proposed in this embodiment.

[0154] First, a semantic feature cascade mechanism was designed to fully integrate multi-dimensional features and gradually enhance their expressive power. Ablation experiments using direct concatenation and progressive fusion validated the advantages of progressive fusion in maintaining semantic consistency and detail integrity. The feature extraction network MobileNetV3 was improved by incorporating a CBAM attention mechanism, employing dynamic convolution kernel adaptation, and designing a parameterized kernel generator to replace the original static convolution. Secondly, a method based on multi-attribute collaborative learning was proposed, which constrains the reconstruction results at both the geometric and semantic levels, significantly improving reconstruction accuracy and stability.

[0155] In addition, the applicant conducted quantitative experiments and visual analysis in different scenarios on the 3D face geometry reconstruction method based on semantic feature cascade and multi-attribute collaborative learning on the Now and AFLW2000-3D datasets. The extensive experimental results show that this method outperforms other similar methods in terms of accuracy, demonstrating the superiority of this method in face reconstruction effects.

[0156] In an exemplary embodiment, Figure 7 As shown, this embodiment provides a 3D face reconstruction system. The system can be deployed on a server, personal computer, or cloud platform in terms of hardware, including a processor, memory, and network interface. At the software level, the system includes the following functional modules:

[0157] Determination module 100, determining a single two-dimensional face image;

[0158] The calling module 200 calls a three-dimensional face reconstruction model based on the two-dimensional face image to generate a three-dimensional face model; wherein,

[0159] The three-dimensional face reconstruction model includes a multi-attribute collaborative learning module, which is used to optimize the initial geometric parameters to obtain target geometric parameters; during the optimization,

[0160] Through the multi-layer perceptron projection layer, the coordinates of the three-dimensional key points are mapped to the geometric parameter space as geometric priors to generate corrections to the initial geometric parameters; the target geometric parameters are the superposition of the initial geometric parameters and the corrections, and the initial geometric parameters and the coordinates of the three-dimensional key points are determined by the two-dimensional face image.

[0161] Optionally, the three-dimensional face reconstruction model further includes a semantic feature cascade module; wherein,

[0162] The semantic feature cascade module is used to encode the two-dimensional face image to obtain a fusion feature of global and local features; the initial geometric parameters and the coordinates of the three-dimensional key points are decoded from the fusion feature.

[0163] Optionally, the semantic feature cascade module includes a backbone network, and the backbone network includes a feature extraction unit;

[0164] The feature extraction unit includes two or more bottleneck layers, and some bottleneck layers include an attention mechanism based on convolution blocks.

[0165] Optionally, the static convolution kernel in the bottleneck layer is replaced by a parameterized kernel generator.

[0166] Optionally, in the process of training the three-dimensional face reconstruction model, for the multi-attribute collaborative learning module, the training key point parameters and the training geometric parameters form a bidirectional constraint through a differentiable mapping matrix; wherein, the bidirectional constraint includes a forward constraint and a reverse gradient, and the training key point parameters and the training geometric parameters are determined by the training images in the training image data set.

[0167] Optionally, the loss function corresponding to the reverse gradient is:

[0168]

[0169] in, is the face mesh generated by training geometric parameters, To train the geometric parameters, is the differentiable mapping matrix; k are the coordinates of the training key points.

[0170] Optionally, the reverse gradient refers to feeding back the calculated gradient to the multi-attribute collaborative learning module to drive the iterative optimization of the training geometric parameters on the Riemannian manifold; wherein the gradient is calculated by the following formula:

[0171]

[0172] in, Represents the loss function A vector of partial derivatives with respect to the model parameters.

[0173] Optionally, the loss function corresponding to the forward constraint is:

[0174]

[0175] in, is the geometric reconstruction loss, To train the geometric parameters, G A face mesh generated by the training geometric parameters; is the key point guidance loss, To train key point parameters; and To balance the weight.

[0176] The system realizes a three-dimensional face reconstruction method through the collaborative work of the above modules.

[0177] In the embodiments of the present application, a 3D face reconstruction system can be a device, or a component, integrated circuit, or chip in a terminal. The device can be a mobile electronic device or a non-mobile electronic device. For example, the mobile electronic device can be a mobile phone, tablet computer, laptop computer, PDA, in-vehicle electronic device, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), while the non-mobile electronic device can be a server, network attached storage (NAS), personal computer (PC), etc., without specific limitation in the embodiments of the present application.

[0178] In the embodiments of the present application, a 3D face reconstruction system may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiments of the present application.

[0179] The three-dimensional face reconstruction system provided in the embodiment of the present application can implement each process implemented by the above-mentioned three-dimensional face reconstruction method. To avoid repetition, they will not be described here.

[0180] Optionally, an embodiment of the present application also provides an electronic device, including a processor, a memory, and a program or instruction stored in the memory and executable on the processor. When the program or instruction is executed by the processor, the various processes of the above-mentioned three-dimensional face reconstruction method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, they will not be described here.

[0181] An embodiment of the present application also provides a readable storage medium, which stores a program or instruction. When the program or instruction is executed by a processor, the various processes of the embodiment of the above-mentioned three-dimensional face reconstruction method are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0182] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer-readable storage medium, such as a computer read-only memory (ROM), random access memory (RAM), a magnetic disk, or an optical disk.

[0183] It should be noted that, in this article, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element. In addition, it should be noted that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in the opposite order according to the functions involved. For example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0184] Through the description of the above embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform. Of course, they can also be implemented by hardware, but in many cases the former is a more preferred embodiment. Based on this understanding, the technical solution of this application, or the part that contributes to the existing technology, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of this application.

[0185] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of this application, ordinary technicians in this field can also make many forms without departing from the purpose of this application and the scope of protection of the claims, all of which are within the protection of this application.

Claims

1. A three-dimensional face reconstruction method, characterized in that: The three-dimensional face reconstruction method comprises: Determine a single two-dimensional face image; Based on the two-dimensional face image, a three-dimensional face reconstruction model is called to generate a three-dimensional face model; wherein, The three-dimensional face reconstruction model includes a multi-attribute collaborative learning module, which is used to optimize the initial geometric parameters to obtain target geometric parameters; during the optimization, The coordinates of the three-dimensional key points are mapped as geometric priors to a geometric parameter space through a multi-layer perceptron projection layer to generate corrections to the initial geometric parameters; the target geometric parameters are the superposition of the initial geometric parameters and the corrections, and the initial geometric parameters and the coordinates of the three-dimensional key points are determined by the two-dimensional face image; The three-dimensional face reconstruction model also includes a semantic feature cascade module; wherein, The semantic feature cascade module is used to encode the two-dimensional face image to obtain a fusion feature of global and local features; the initial geometric parameters and the coordinates of the three-dimensional key points are decoded from the fusion feature; The semantic feature cascade module includes a backbone network, and the backbone network includes a feature extraction unit; The feature extraction unit includes two or more bottleneck layers, and some bottleneck layers include an attention mechanism based on convolution blocks.

2. The 3D face reconstruction method according to claim 1, wherein: The static convolution kernel in the bottleneck layer is replaced by a parameterized kernel generator.

3. The three-dimensional face reconstruction method according to claim 1 or 2, characterized in that: In the process of training the three-dimensional face reconstruction model, for the multi-attribute collaborative learning module, the training key point parameters and the training geometric parameters form a bidirectional constraint through a differentiable mapping matrix; wherein, the bidirectional constraint includes a forward constraint and a reverse gradient, and the training key point parameters and the training geometric parameters are determined by the training images in the training image data set.

4. The 3D face reconstruction method according to claim 3, wherein: The loss function corresponding to the reverse gradient is: in, is the face mesh generated by training geometric parameters, To train the geometric parameters, is the differentiable mapping matrix; are the coordinates of the training key points.

5. The 3D face reconstruction method according to claim 4, characterized in that: The reverse gradient refers to feeding back the calculated gradient to the multi-attribute collaborative learning module to drive the iterative optimization of the training geometric parameters on the Riemannian manifold; wherein the gradient is calculated by the following formula: in, Represents the loss function A vector of partial derivatives with respect to the model parameters.

6. The 3D face reconstruction method according to claim 3, wherein: The loss function corresponding to the forward constraint is: in, is the geometric reconstruction loss, To train the geometric parameters, A face mesh generated by the training geometric parameters; is the key point guidance loss, To train key point parameters; and To balance the weight.

7. An electronic device, characterized in that: The method comprises a processor, a memory, and a program or instruction stored in the memory and executable on the processor. When the program or instruction is executed by the processor, the steps of the three-dimensional face reconstruction method according to any one of claims 1 to 6 are implemented.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a program or instruction, and when the program or instruction is executed by the processor, the steps of the three-dimensional face reconstruction method according to any one of claims 1 to 6 are implemented.

Images