A method, apparatus and equipment for three-dimensional face reconstruction and six-degree-of-freedom pose estimation

Abstract

This invention provides a method, apparatus, and device for 3D face reconstruction and six-DOF pose estimation. The method includes: acquiring a face image to be reconstructed; inputting the face image to be reconstructed into a 3D face reconstruction model, obtaining and outputting the 3D reconstruction result of the face image to be reconstructed and the six-DOF pose of the face in the face image to be reconstructed from the 3D face reconstruction model; the 3D face reconstruction model is trained based on the 3D reconstruction prediction result of a sample face image and the labeled 3D reconstruction result of the sample face image, and the six-DOF pose of the sample face image is constrained based on the correlation matrix between the 3D reconstruction prediction result and the two-dimensional pixels of the face portion. The method, apparatus, and device provided by this invention improve the accuracy of face pose estimation, further improve the accuracy and reliability of face reconstruction, and enhance the robustness of face reconstruction.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a method, apparatus and device for three-dimensional face reconstruction and six-degree-of-freedom pose estimation. Background Technology

[0002] With the development of deep learning, technologies such as virtual wearables, virtual makeup, video editing, animation production, and fatigue recognition have been continuously improved and perfected, and the application needs of many mobile phones and computers have also been increasing.

[0003] Affine transformation-based face reconstruction methods suffer from instability due to distortions in the image caused by changes in facial pose. Furthermore, previous work on facial pose often only focuses on the three Euler angles of the face—pitch, yaw, and roll—ignoring the face's offset within the image, and the resulting rotation angle parameters are not precise enough to meet the needs of AR (Virtual Reality) / VR (Augmented Reality) applications. For example, in virtual glasses try-on applications, inaccurate head pose estimation leads to inaccurate projection of the glasses onto the head; here, the required pose is not just the three rotation angles, but also the face's offset within the image. In recent virtual makeup application developments, the fitting channel requires precise facial pose to achieve pixel-level overlap of the virtual makeup on the face, a task that previous methods are also insufficient for. Summary of the Invention

[0004] This invention provides a method, apparatus, and device for three-dimensional face reconstruction and six-degree-of-freedom pose estimation, which solves the defect of existing face reconstruction methods based on affine transformations, which suffer from distortion in the image due to changes in face pose, leading to unstable reconstruction.

[0005] This invention provides a method for three-dimensional face reconstruction and six-degree-of-freedom pose estimation, including:

[0006] Obtain the image of the face to be reconstructed;

[0007] The face image to be reconstructed is input into a three-dimensional face reconstruction model, and the three-dimensional reconstruction result of the face image to be reconstructed and the six-degree-of-freedom pose of the face in the face image to be reconstructed are obtained and output by the three-dimensional face reconstruction model.

[0008] The three-dimensional face reconstruction model is trained based on the three-dimensional reconstruction prediction results of the sample face image and the labeled three-dimensional reconstruction results of the sample face image, and is obtained by constraining the six-degree-of-freedom pose of the sample face image based on the correlation matrix between the three-dimensional reconstruction prediction results and the two-dimensional pixels of the face part.

[0009] According to the present invention, a method for three-dimensional face reconstruction and six-degree-of-freedom pose estimation is provided, wherein the training steps of the three-dimensional face reconstruction model include:

[0010] Acquire the first sample face image, the second sample face image containing only the face, the true six-DOF pose of the face, the label 3D reconstruction result, the label association matrix, the label 3D coordinates corresponding to each point, and the initial 3D face reconstruction model;

[0011] The initial 3D face reconstruction model and the actual six-DOF pose of the face are projected onto the image to obtain the face position coordinate annotation;

[0012] Determine the position coordinates of the face in the second sample face image;

[0013] Extract the global features of the face image of the first sample face image, and the three-dimensional features and local features of the face image of the second sample face image;

[0014] Based on the three-dimensional features corresponding to the second sample face image, the three-dimensional reconstruction prediction result of the second sample face image is determined;

[0015] Based on the global features and local features of the face image, the correlation matrix between the 3D reconstruction prediction result and the 2D pixels of the face portion is determined;

[0016] Based on the correlation matrix and the 3D reconstruction prediction results, the predicted 3D point coordinates corresponding to each 2D pixel are obtained;

[0017] Based on the 3D reconstruction prediction results and the labeled 3D reconstruction results, the face location coordinates and the location coordinates, the correlation matrix and the label correlation matrix, the predicted 3D point coordinates corresponding to each 2D pixel and the label 3D coordinates corresponding to each point, the parameters of the initial 3D face reconstruction model are iterated to obtain the 3D face reconstruction model.

[0018] According to the present invention, a three-dimensional face reconstruction and six-degree-of-freedom pose estimation method is provided, wherein based on the three-dimensional reconstruction prediction result and the labeled three-dimensional reconstruction result, the face position coordinate annotation and the position coordinate, the correlation matrix and the label correlation matrix, the predicted three-dimensional point coordinates corresponding to each two-dimensional pixel and the labeled three-dimensional coordinates corresponding to each point, the initial three-dimensional face reconstruction model is iterated to obtain a three-dimensional face reconstruction model, including:

[0019] Based on the 3D reconstruction prediction results and the label 3D reconstruction results, the first loss is determined;

[0020] Based on the face location coordinates and the location coordinates, a second loss is determined;

[0021] Based on the correlation matrix and the label correlation matrix, a third loss is determined;

[0022] Based on the predicted 3D point coordinates corresponding to each 2D pixel and the label 3D coordinates corresponding to each point, the fourth loss is determined.

[0023] Based on the first loss, the second loss, the third loss, and the fourth loss, the parameters of the initial 3D face reconstruction model are iterated to obtain the 3D face reconstruction model.

[0024] According to the present invention, a method for three-dimensional face reconstruction and six-degree-of-freedom pose estimation is provided, wherein determining the three-dimensional reconstruction prediction result of the second sample face image based on the three-dimensional features corresponding to the second sample face image includes:

[0025] Based on the aforementioned three-dimensional features, the UV location map of the face is determined;

[0026] The UV location map of the face is sequentially subjected to UV pairing and mesh sampling to obtain the three-dimensional reconstruction prediction result.

[0027] According to the present invention, a method for three-dimensional face reconstruction and six-degree-of-freedom pose estimation is provided, wherein determining the correlation matrix between the three-dimensional reconstruction prediction result and the two-dimensional pixels of the face portion based on the global features and the local features of the face image includes:

[0028] Two-dimensional point coordinates of the face are sampled from the position coordinates of the face in the second sample face image;

[0029] Based on the mapping from the second sample face image to the first sample face image, the full... Figure 2 3D point coordinates;

[0030] Two-dimensional local features are determined based on the two-dimensional point coordinates of the face, and two-dimensional local features are determined based on the full face coordinates. Figure 2 Two-dimensional point coordinates determine two-dimensional global features;

[0031] Based on the 3D reconstruction prediction results, 3D local features are determined, and based on the 3D local features and the multilayer perceptron, 3D global features are obtained.

[0032] Based on the Transformer model, the two-dimensional local features, the two-dimensional global features, the three-dimensional local features, and the three-dimensional global features are fused to obtain the correlation matrix between the three-dimensional reconstruction prediction result and the two-dimensional pixels of the face.

[0033] According to the present invention, a method for three-dimensional face reconstruction and six-degree-of-freedom pose estimation is provided, wherein the predicted three-dimensional point coordinates corresponding to each two-dimensional pixel are obtained based on the correlation matrix and the three-dimensional reconstruction prediction results, and the method further includes:

[0034] Based on the coordinates of each two-dimensional pixel and the corresponding predicted three-dimensional point, pose estimation is performed to obtain the predicted six-degree-of-freedom pose of the face.

[0035] According to the present invention, a method for three-dimensional face reconstruction and six-degree-of-freedom pose estimation is provided, wherein determining the position coordinates of the face in the second sample face image includes:

[0036] The second sample face image is input into the face segmentation model, which obtains and outputs the position coordinates corresponding to the face; the face segmentation model is built based on the ResNet model.

[0037] The present invention also provides a three-dimensional face reconstruction and six-degree-of-freedom pose estimation device, comprising:

[0038] The acquisition unit is used to acquire the face image to be reconstructed.

[0039] The three-dimensional reconstruction unit is used to input the face image to be reconstructed into the three-dimensional face reconstruction model, and to obtain and output the three-dimensional reconstruction result of the face image to be reconstructed and the six-degree-of-freedom pose of the face in the face image to be reconstructed by the three-dimensional face reconstruction model.

[0040] The three-dimensional face reconstruction model is trained based on the three-dimensional reconstruction prediction results of the sample face image and the labeled three-dimensional reconstruction results of the sample face image, and is obtained by constraining the six-degree-of-freedom pose of the sample face image based on the correlation matrix between the three-dimensional reconstruction prediction results and the two-dimensional pixels of the face part.

[0041] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the three-dimensional face reconstruction and six-degree-of-freedom pose estimation method as described above.

[0042] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the three-dimensional face reconstruction and six-degree-of-freedom pose estimation method as described above.

[0043] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the three-dimensional face reconstruction and six-degree-of-freedom pose estimation method as described above.

[0044] The present invention provides a method, apparatus, and device for three-dimensional face reconstruction and six-degree-of-freedom pose estimation. The three-dimensional face reconstruction model is trained based on the three-dimensional reconstruction prediction results of sample face images and the labeled three-dimensional reconstruction results of sample face images. The model is obtained by constraining the six-degree-of-freedom pose of sample face images based on the correlation matrix between the three-dimensional reconstruction prediction results and the two-dimensional pixels of the face portion. This improves the accuracy of face pose and further enhances the accuracy and reliability of face reconstruction. Attached Figure Description

[0045] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0046] Figure 1 This is a flowchart illustrating the three-dimensional face reconstruction and six-degree-of-freedom pose estimation method provided by the present invention.

[0047] Figure 2 This is a flowchart illustrating the training steps of the three-dimensional face reconstruction model provided by the present invention.

[0048] Figure 3 This is a flowchart illustrating step 280 in the three-dimensional face reconstruction and six-degree-of-freedom pose estimation method provided by the present invention.

[0049] Figure 4 This is a flowchart illustrating step 250 in the three-dimensional face reconstruction and six-degree-of-freedom pose estimation method provided by the present invention.

[0050] Figure 5 This is a flowchart illustrating step 260 in the three-dimensional face reconstruction and six-degree-of-freedom pose estimation method provided by the present invention.

[0051] Figure 6 This is a schematic diagram of the structure of the three-dimensional face reconstruction and six-degree-of-freedom pose estimation device provided by the present invention;

[0052] Figure 7 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0054] The terms "first," "second," etc., used in the specification and claims of this invention are used to distinguish similar objects and are not used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and that the objects distinguished by "first," "second," etc., are generally of the same class.

[0055] This invention provides a method for three-dimensional face reconstruction and six-degree-of-freedom pose estimation. Figure 1 This is a flowchart illustrating the three-dimensional face reconstruction and six-degree-of-freedom pose estimation method provided by the present invention, as shown below. Figure 1 As shown, the method includes:

[0056] Step 110: Obtain the face image to be reconstructed.

[0057] Specifically, the face image to be reconstructed, i.e. the image for which 3D face reconstruction is subsequently performed, can be pre-acquired by an image acquisition device, captured in real time, or downloaded or scanned via the Internet. This embodiment of the invention does not impose specific limitations on these aspects.

[0058] Step 120: Input the face image to be reconstructed into the three-dimensional face reconstruction model, and obtain and output the three-dimensional reconstruction result of the face image to be reconstructed and the six-degree-of-freedom pose of the face in the face image to be reconstructed from the three-dimensional face reconstruction model.

[0059] The three-dimensional face reconstruction model is trained based on the three-dimensional reconstruction prediction results of the sample face image and the labeled three-dimensional reconstruction results of the sample face image, and is obtained by constraining the six-degree-of-freedom pose of the sample face image based on the correlation matrix between the three-dimensional reconstruction prediction results and the two-dimensional pixels of the face part.

[0060] Specifically, in order to obtain better 3D reconstruction results and the six-DOF pose of the face, a 3D face reconstruction model needs to be obtained through the following steps before step 120:

[0061] It can collect sample face images and labeled 3D reconstruction results of sample face images in advance, and can also build an initial 3D face reconstruction model in advance.

[0062] In this process, the sample face image can be input into the initial 3D face reconstruction model, and the initial 3D face reconstruction model can obtain and output the 3D reconstruction prediction result of the sample face image. During the training process, the six degrees of freedom pose of the sample face image is constrained based on the correlation matrix between the 3D reconstruction prediction result and the two-dimensional pixels of the face part. The six degrees of freedom pose here includes 3D position information and face pose information.

[0063] Here, the correlation matrix between each 3D reconstruction prediction result and the 2D pixels of the face is labeled as matrix M. If the i-th pixel is at face vertex P... i P j P k On the triangular facet formed by these three vertices, the probability of pixel i corresponding to these three vertices is w. i w j w k That is, the i-th row and i-th column, i-th row and j-th column, and i-th row and k-th column of matrix M is w. i w j w k The rest are set to 0.

[0064] It should be noted that w i w j w k The sum of is 1.

[0065] The correlation matrix between the 3D reconstruction prediction result and the 2D pixels of the face portion can be determined based on the global features of the face image corresponding to the sample face image and the local features corresponding to the cropped image of the sample face image that only includes the face.

[0066] After obtaining the 3D reconstruction prediction result based on the initial 3D face reconstruction model, the 3D reconstruction prediction result can be compared with the labeled 3D reconstruction results of the pre-collected sample face images. The loss function value is calculated based on the degree of difference between the two, and the parameters of the initial 3D face reconstruction model are iterated based on the loss function value. During the training process, the six-degree-of-freedom pose of the sample face image is regressed based on the correlation matrix between the 3D reconstruction prediction result and the two-dimensional pixels of the face part. The initial 3D face reconstruction model after parameter iteration is denoted as the 3D face reconstruction model.

[0067] Understandably, the greater the difference between the 3D reconstruction prediction result and the labeled 3D reconstruction result of the pre-collected sample face images, the larger the loss function value; conversely, the smaller the difference between the 3D reconstruction prediction result and the labeled 3D reconstruction result of the pre-collected sample face images, the smaller the loss function value.

[0068] During the training process of the 3D face reconstruction model, it learned the function of obtaining and outputting the 3D reconstruction result of the face image to be reconstructed and the six-degree-of-freedom pose of the face in the face image to be reconstructed.

[0069] The method provided in this invention uses a three-dimensional face reconstruction model trained based on the three-dimensional reconstruction prediction results of sample face images and the labeled three-dimensional reconstruction results of sample face images. It also constrains the six-degree-of-freedom pose of the sample face image based on the correlation matrix between the three-dimensional reconstruction prediction results and the two-dimensional pixels of the face part. This improves the accuracy of face pose and further enhances the accuracy and reliability of face reconstruction.

[0070] Based on the above embodiments, Figure 2 This is a flowchart illustrating the training steps of the 3D face reconstruction model provided by the present invention, as shown below. Figure 2 As shown, the training steps of the 3D face reconstruction model include:

[0071] Step 210: Obtain the first sample face image, the second sample face image containing only the face, the true six-degree-of-freedom pose of the face, the label 3D reconstruction result, the label association matrix, the label 3D coordinates corresponding to each point, and the initial 3D face reconstruction model.

[0072] Step 220: Project the initial 3D face reconstruction model and the true six-degree-of-freedom pose of the face into the image to obtain the face position coordinate annotation;

[0073] Step 230: Determine the position coordinates of the face in the second sample face image;

[0074] Step 240: Extract the global features of the face image of the first sample face image, and the three-dimensional features and local features corresponding to the face image of the second sample face image;

[0075] Step 250: Based on the three-dimensional features corresponding to the second sample face image, determine the three-dimensional reconstruction prediction result of the second sample face image;

[0076] Step 260: Based on the global features and local features of the face image, determine the correlation matrix between the 3D reconstruction prediction result and the 2D pixels of the face portion;

[0077] Step 270: Based on the correlation matrix and the 3D reconstruction prediction results, obtain the predicted 3D point coordinates corresponding to each 2D pixel;

[0078] Step 280: Based on the 3D reconstruction prediction results and the label 3D reconstruction results, the face location coordinate annotation and the location coordinates, the correlation matrix and the label correlation matrix, the predicted 3D point coordinates corresponding to each 2D pixel and the label 3D coordinates corresponding to each point, perform parameter iteration on the initial 3D face reconstruction model to obtain the 3D face reconstruction model.

[0079] Specifically, in order to obtain a better 3D face reconstruction model, the 3D face reconstruction model can be trained through the following steps:

[0080] It can acquire a first sample face image, a second sample face image containing only the face, the true six-DOF pose of the face, the labeled 3D reconstruction results, the label association matrix, the label 3D coordinates corresponding to each point, and the initial 3D face reconstruction model. The first sample face image, the second sample face image containing only the face, and the true six-DOF pose of the face are all determined based on the scanning model and its camera parameters. When the scanning model scans a face image, the coordinate system of the face scanning model is the world coordinate system, and the camera projection parameters are determined according to the scanning model. The face scanning model is obtained by scanning a 3D face using a scanner.

[0081] Then, the initial 3D face reconstruction model and the true six-DOF pose of the face are projected onto the image to obtain the face position coordinates. For example, the initial 3D face reconstruction model is projected onto a black background image using the true six-DOF pose of the face and camera parameters, with the face portion set to white. For the original 800*800 image, a solid-color background image with all pixel values ​​set to 255 is preset. After projecting the initial 3D face reconstruction model onto the image using the true six-DOF pose of the face and camera parameters, the covered portion is set to 0, resulting in a binary face segmentation location map, i.e., the face position coordinates. This process uses the camera parameters as intrinsic camera parameters and the true six-DOF pose of the face as extrinsic camera parameters for projection. The camera projection parameters are determined based on the scanning model.

[0082] The initial 3D face reconstruction model is projected onto the image using the face's true six-DOF pose and camera parameters. Each triangle composed of three face vertices contains some face pixels. The probability of a pixel corresponding to a face vertex is determined by the centroid coordinates of the triangle and the pixel coordinates.

[0083] It should be noted that each two-dimensional pixel has a probability of corresponding with only three vertices of the triangle it is in, and the probability of corresponding with the other vertices is 0.

[0084] Specifically, each correlation matrix is ​​labeled as matrix M, if the i-th pixel is at face vertex P i P j Pk On the triangular facet formed by these three vertices, the probability of pixel i corresponding to these three vertices is w. i w j w k That is, the i-th row and i-th column, i-th row and j-th column, and i-th row and k-th column of matrix M is w. i w j w k The rest are set to 0.

[0085] It should be noted that w i w j w k The sum of is 1.

[0086] Next, determine the position coordinates of the face in the second sample face image. The second sample face image can be obtained by cropping the face detected from the first sample face image using a face detection model.

[0087] For example, a second sample face image containing only the face can be input into a face segmentation model to segment the face and obtain the position coordinates of the face in the second sample face image.

[0088] Furthermore, global features of the face image of the first sample face image and three-dimensional features and local features corresponding to the second sample face image can be extracted. Here, the first sample face image and the second sample face image can be input into the feature extraction model to obtain global features of the face image of the first sample face image and three-dimensional features and local features corresponding to the second sample face image.

[0089] The feature extraction model here can be a cascaded multilayer convolutional neural network (CNN), a deep neural network (DNN), or a ResNet model, etc. This embodiment of the invention does not specifically limit it.

[0090] For example, an encoder based on a ResNet model is constructed to extract features from the input image, and then a decoder is constructed by stacking deconvolution layers to decode the image features, obtaining 3D features, global features of the face image, and local features respectively.

[0091] It should be noted that the encoder of the feature extraction model is built using the ResNet model, and can be shared when extracting 3D features and local features. The parameters after model training can also be shared. However, the decoder only has the same structure and cannot share parameters.

[0092] The global features of the face image here reflect the global feature information of the first sample face image, the three-dimensional features here reflect the three-dimensional feature information of the second sample face image, and the local features here reflect the local feature information of the second sample face image.

[0093] Then, based on the three-dimensional features corresponding to the second sample face image, the three-dimensional reconstruction prediction result of the second sample face image is determined. For example, the UV location map of the face can be determined based on the three-dimensional features corresponding to the second sample face image, and then the three-dimensional reconstruction prediction result of the second sample face image can be obtained based on the UV location map.

[0094] Then, based on the global and local features of the face image, the correlation matrix between the 3D reconstruction prediction result and the 2D pixels of the face is determined. This can be achieved based on the Transformer model.

[0095] Then, based on the correlation matrix and the 3D reconstruction prediction results, the predicted 3D point coordinates corresponding to each 2D pixel are obtained.

[0096] Finally, after obtaining the 3D reconstruction prediction results and labeled 3D reconstruction results, face location coordinates and location coordinates, correlation matrix and label correlation matrix, predicted 3D point coordinates corresponding to each 2D pixel and labeled 3D coordinates corresponding to each point, the initial 3D face reconstruction model can be iterated to obtain the 3D face reconstruction model based on the 3D reconstruction prediction results and labeled 3D reconstruction results, face location coordinates and location coordinates, correlation matrix and label correlation matrix, predicted 3D point coordinates corresponding to each 2D pixel and labeled 3D coordinates corresponding to each point.

[0097] Based on the above embodiments, Figure 3 This is a flowchart illustrating step 280 in the three-dimensional face reconstruction and six-DOF pose estimation method provided by the present invention, as shown below. Figure 3 As shown, step 280 includes:

[0098] Step 281: Based on the 3D reconstruction prediction results and the label 3D reconstruction results, determine the first loss;

[0099] Step 282: Determine the second loss based on the face location coordinate annotation and the location coordinates;

[0100] Step 283: Determine the third loss based on the correlation matrix and the label correlation matrix;

[0101] Step 284: Determine the fourth loss based on the predicted three-dimensional point coordinates corresponding to each two-dimensional pixel and the label three-dimensional coordinates corresponding to each point;

[0102] Step 285: Based on the first loss, the second loss, the third loss, and the fourth loss, perform parameter iteration on the initial 3D face reconstruction model to obtain the 3D face reconstruction model.

[0103] Specifically, the first loss can be determined based on the 3D reconstruction prediction results and the label 3D reconstruction results, using the UV location map to replace the 3D points. The specific calculation formula for the first loss is as follows:

[0104] L r =W·∑||U * -U||

[0105] Among them, U * It is a UV location map generated based on the 3D reconstruction results of the label. U is the location map predicted by the model, and W is a weight matrix with the same size as the UV location map.

[0106] Understandably, the greater the difference between the 3D reconstruction prediction result and the labeled 3D reconstruction result, the greater the first loss; the smaller the difference between the 3D reconstruction prediction result and the labeled 3D reconstruction result, the smaller the first loss.

[0107] The second loss can be determined based on the face location coordinates and location coordinates.

[0108] It is understandable that the greater the difference between the face location coordinates and the actual location coordinates, the greater the second loss; the smaller the difference between the face location coordinates and the actual location coordinates, the smaller the second loss.

[0109] The third loss can be determined based on the correlation matrix and the label correlation matrix. The formula for the third loss is as follows:

[0110]

[0111] Among them, M and M * These are the predicted association matrix and the label association matrix, respectively, where λ is the weight, m ​​is the number of 2D pixels, n is the number of 3D points, i represents the i-th row, j represents the j-th column, and D... KL This represents the KL divergence loss.

[0112] It is understandable that the greater the difference between the association matrix and the label association matrix, the greater the third loss; the smaller the difference between the association matrix and the label association matrix, the smaller the third loss.

[0113] The fourth loss can be determined based on the predicted 3D point coordinates corresponding to each 2D pixel and the corresponding label 3D coordinates. The formula for the fourth loss is as follows:

[0114] Lc =||X 3d -X 3d* ||1

[0115] Among them, X 3d X represents the predicted 3D point coordinates corresponding to each 2D pixel. 3d* This represents the three-dimensional coordinates of the label corresponding to each point.

[0116] It is understandable that the greater the difference between the predicted 3D point coordinates corresponding to each 2D pixel and the corresponding label 3D coordinates, the greater the fourth loss; the smaller the difference between the predicted 3D point coordinates corresponding to each 2D pixel and the corresponding label 3D coordinates, the smaller the fourth loss.

[0117] After obtaining the first loss, second loss, third loss, and fourth loss, the parameters of the initial 3D face reconstruction model can be iterated based on the first loss, second loss, third loss, and fourth loss to obtain the 3D face reconstruction model.

[0118] Here, the initial 3D face reconstruction model can be iterated based on the sum of the first loss, the second loss, the third loss, and the fourth loss, or based on the weighted sum of the first loss, the second loss, the third loss, and the fourth loss, to obtain the 3D face reconstruction model.

[0119] The formula for determining the total loss based on the weighted sum of the first loss, second loss, third loss, and fourth loss is as follows:

[0120] L=λ1L r +λ2L s +λ3L m +λ4L c

[0121] Where L represents the total loss, and λ1, λ2, λ3, and λ4 are the weight parameters corresponding to the first loss, second loss, third loss, and fourth loss, respectively. r As the first loss, L s For the second loss, L m As the third loss, L c This is the fourth loss.

[0122] The method provided in this embodiment of the invention optimizes the parameters of the initial 3D face reconstruction model based on the first loss, the second loss, the third loss, and the fourth loss, and determines whether the total loss converges. If it converges, the training is terminated and the parameters of the 3D face reconstruction model are obtained; otherwise, the above method is repeated until a trained 3D face reconstruction model is obtained, thereby improving the reconstruction accuracy of the 3D face reconstruction model.

[0123] Based on the above embodiments, Figure 4This is a flowchart illustrating step 250 in the three-dimensional face reconstruction and six-DOF pose estimation method provided by the present invention, as shown below. Figure 4 As shown, step 250 includes:

[0124] Step 251: Based on the three-dimensional features, determine the UV location map of the face;

[0125] Step 252: Perform UV pairing and mesh sampling on the UV location map of the face in sequence to obtain the three-dimensional reconstruction prediction result.

[0126] Specifically, the three-dimensional features, i.e. the three-dimensional feature map, can be used as the UV location map of the face. After obtaining the UV location map of the face, UV matching (UV Coordinates) and grid sampling (GridSample) can be performed on the UV location map of the face in sequence to obtain the three-dimensional reconstruction prediction result.

[0127] That is, the 3D reconstruction prediction results are obtained by regressing the UV location map through a neural network, which prepares for subsequent association learning and pose estimation.

[0128] Based on the above embodiments, Figure 5 This is a flowchart illustrating step 260 in the three-dimensional face reconstruction and six-DOF pose estimation method provided by the present invention, as shown below. Figure 5 As shown, step 260 includes:

[0129] Step 261: Sample the two-dimensional point coordinates of the face from the position coordinates of the face in the second sample face image;

[0130] Step 262: Based on the mapping from the second sample face image to the first sample face image, obtain the full... Figure 2 3D point coordinates;

[0131] Step 263: Determine two-dimensional local features based on the two-dimensional point coordinates of the face, and determine the full-dimensional features based on the two-dimensional point coordinates of the face. Figure 2 Two-dimensional point coordinates determine two-dimensional global features;

[0132] Step 264: Based on the 3D reconstruction prediction results, determine the 3D local features, and based on the 3D local features and the multilayer perceptron, obtain the 3D global features;

[0133] Step 265: Based on the Transformer model, the two-dimensional local features, the two-dimensional global features, the three-dimensional local features, and the three-dimensional global features are fused to obtain the correlation matrix between the three-dimensional reconstruction prediction result and the two-dimensional pixels of the face.

[0134] Specifically, the two-dimensional point coordinates of the face can be sampled from the position coordinates of the face in the second sample face image. These two-dimensional point coordinates reflect the two-dimensional position information of the face in the second sample face image.

[0135] Based on the mapping from the second sample face image to the first sample face image, the full... Figure 2 3D point coordinates, here the whole Figure 2 The point coordinates reflect the two-dimensional positional information of the entire image in the first sample face image.

[0136] Two-dimensional local features can be determined based on the coordinates of two-dimensional points on a face, and based on the full... Figure 2 Two-dimensional point coordinates determine two-dimensional global features, while two-dimensional local features reflect local feature information at the two-dimensional level, and two-dimensional global features reflect global feature information at the two-dimensional level.

[0137] Then, based on the 3D reconstruction prediction results, 3D local features can be determined, and based on the 3D local features and the multilayer perceptron, 3D global features can be obtained.

[0138] The three-dimensional local features here reflect local feature information at the three-dimensional level, while the three-dimensional global features here reflect global feature information at the three-dimensional level.

[0139] Here, three-dimensional local features can be mapped to three-dimensional global features based on a multi-layer perceptron (MLP).

[0140] After obtaining the two-dimensional local features, two-dimensional global features, three-dimensional local features, and three-dimensional global features, the Transformer model can be used to fuse these features to obtain the correlation matrix between the three-dimensional reconstruction prediction result and the two-dimensional pixels of the face. The formula for the correlation matrix is ​​as follows:

[0141] M = T(f)

[0142] Where T is the Transformer model with attention mechanism, f is the fusion feature of two-dimensional local features, two-dimensional global features, three-dimensional local features and three-dimensional global features, and M is the learned correlation matrix.

[0143] Understandably, Transformer models with self-attention mechanisms can handle fused features well, making the learned correlation matrix more accurate.

[0144] The above scheme inputs a face image into a trained 3D face reconstruction model, performs forward computation, extracts global and local features of the face, and combines these with prior knowledge of the 3D face shape to reconstruct a 3D face reconstruction prediction result. It also learns the correspondence between 3D face points and 2D pixels, thereby calculating the six-DOF pose of the face. By using a Transformer model with an attention mechanism to learn and process fused features, it optimizes the correlation matrix learning and improves the robustness of face reconstruction through a perspective projection-based reconstruction method, while also increasing the accuracy of face pose.

[0145] The method provided in this embodiment of the invention is based on the Transformer model, which fuses two-dimensional local features, two-dimensional global features, three-dimensional local features and three-dimensional global features to obtain the correlation matrix between the three-dimensional reconstruction prediction result and the two-dimensional pixels of the face part, thereby improving the accuracy of the correlation matrix learning and making the subsequent pose calculation more accurate.

[0146] Based on the above embodiments, step 270 further includes:

[0147] Step 271: Based on each two-dimensional pixel and the predicted three-dimensional point coordinates corresponding to each two-dimensional pixel, pose estimation is performed to obtain the predicted six-degree-of-freedom pose of the face.

[0148] Specifically, pose estimation is performed based on the coordinates of each two-dimensional pixel and the corresponding predicted three-dimensional point to obtain the six-degree-of-freedom pose of the face. The pose estimation here can be the PnP (Perspective-n-Point) algorithm, but this embodiment of the invention does not specifically limit it.

[0149] Based on the above embodiments, step 230 includes:

[0150] The second sample face image is input into the face segmentation model, which obtains and outputs the position coordinates corresponding to the face; the face segmentation model is built based on the ResNet model.

[0151] Specifically, the second sample face image can be input into the face segmentation model, which will then obtain and output the position coordinates of the face. For example, the encoder of the face segmentation model can be constructed using a ResNet model to extract face image features, and then a decoder can be constructed by stacking deconvolution layers to decode the image features and obtain the segmented image, thereby determining the position coordinates of the face.

[0152] The following describes the three-dimensional face reconstruction and six-degree-of-freedom pose estimation device provided by the present invention. The three-dimensional face reconstruction and six-degree-of-freedom pose estimation device described below can be referred to in correspondence with the three-dimensional face reconstruction and six-degree-of-freedom pose estimation method described above.

[0153] Based on any of the above embodiments, the present invention provides a three-dimensional face reconstruction and six-degree-of-freedom pose estimation device. Figure 6 This is a schematic diagram of the structure of the three-dimensional face reconstruction and six-degree-of-freedom pose estimation device provided by the present invention, as shown below. Figure 6 As shown, the device includes:

[0154] Acquisition unit 610 is used to acquire the face image to be reconstructed;

[0155] The three-dimensional reconstruction unit 620 is used to input the face image to be reconstructed into the three-dimensional face reconstruction model, obtain and output the three-dimensional reconstruction result of the face image to be reconstructed and the six-degree-of-freedom pose of the face in the face image to be reconstructed from the three-dimensional face reconstruction model;

[0156] The three-dimensional face reconstruction model is trained based on the three-dimensional reconstruction prediction results of the sample face image and the labeled three-dimensional reconstruction results of the sample face image, and is obtained by constraining the six-degree-of-freedom pose of the sample face image based on the correlation matrix between the three-dimensional reconstruction prediction results and the two-dimensional pixels of the face part.

[0157] The apparatus provided in this invention uses a three-dimensional face reconstruction model trained based on the three-dimensional reconstruction prediction results of sample face images and the labeled three-dimensional reconstruction results of sample face images. It also constrains the six-degree-of-freedom pose of the sample face image based on the correlation matrix between the three-dimensional reconstruction prediction results and the two-dimensional pixels of the face portion. This improves the accuracy of face pose and further enhances the accuracy and reliability of face reconstruction.

[0158] Based on any of the above embodiments, the training steps of the 3D face reconstruction model are specifically used for:

[0159] The sample acquisition unit is used to acquire the first sample face image, the second sample face image containing only the face, the true six-degree-of-freedom pose of the face, the label 3D reconstruction result, the label association matrix, the label 3D coordinates corresponding to each point, and the initial 3D face reconstruction model.

[0160] A face position coordinate annotation unit is used to project the initial three-dimensional face reconstruction model and the true six-degree-of-freedom pose of the face into the image to obtain face position coordinate annotations;

[0161] The location coordinate unit is used to determine the location coordinates of the face in the second sample face image;

[0162] The feature extraction unit is used to extract the global features of the face image of the first sample face image, the three-dimensional features and local features corresponding to the face image of the second sample face image;

[0163] The three-dimensional reconstruction prediction result determination unit is used to determine the three-dimensional reconstruction prediction result of the second sample face image based on the three-dimensional features corresponding to the second sample face image;

[0164] The correlation matrix unit is used to determine the correlation matrix between the three-dimensional reconstruction prediction result and the two-dimensional pixels of the face based on the global features and local features of the face image.

[0165] The predicted 3D point coordinate unit is determined and used to obtain the predicted 3D point coordinates corresponding to each 2D pixel based on the correlation matrix and the 3D reconstruction prediction result;

[0166] The parameter iteration unit is used to perform parameter iteration on the initial 3D face reconstruction model based on the 3D reconstruction prediction result and the label 3D reconstruction result, the face position coordinate annotation and the position coordinate, the correlation matrix and the label correlation matrix, the predicted 3D point coordinates corresponding to each 2D pixel and the label 3D coordinates corresponding to each point, to obtain the 3D face reconstruction model.

[0167] Based on any of the above embodiments, the parameter iteration unit is specifically used for:

[0168] Based on the 3D reconstruction prediction results and the label 3D reconstruction results, the first loss is determined;

[0169] Based on the face location coordinates and the location coordinates, a second loss is determined;

[0170] Based on the correlation matrix and the label correlation matrix, a third loss is determined;

[0171] Based on the predicted 3D point coordinates corresponding to each 2D pixel and the label 3D coordinates corresponding to each point, the fourth loss is determined.

[0172] Based on the first loss, the second loss, the third loss, and the fourth loss, the parameters of the initial 3D face reconstruction model are iterated to obtain the 3D face reconstruction model.

[0173] Based on any of the above embodiments, the three-dimensional reconstruction prediction result unit is specifically used for:

[0174] Based on the aforementioned three-dimensional features, the UV location map of the face is determined;

[0175] The UV location map of the face is sequentially subjected to UV pairing and mesh sampling to obtain the three-dimensional reconstruction prediction result.

[0176] Based on any of the above embodiments, the correlation matrix unit is determined, specifically for:

[0177] Two-dimensional point coordinates of the face are sampled from the position coordinates of the face in the second sample face image;

[0178] Based on the mapping from the second sample face image to the first sample face image, the full... Figure 2 3D point coordinates;

[0179] Two-dimensional local features are determined based on the two-dimensional point coordinates of the face, and two-dimensional local features are determined based on the full face coordinates. Figure 2 Two-dimensional point coordinates determine two-dimensional global features;

[0180] Based on the 3D reconstruction prediction results, 3D local features are determined, and based on the 3D local features and the multilayer perceptron, 3D global features are obtained.

[0181] Based on the Transformer model, the two-dimensional local features, the two-dimensional global features, the three-dimensional local features, and the three-dimensional global features are fused to obtain the correlation matrix between the three-dimensional reconstruction prediction result and the two-dimensional pixels of the face.

[0182] Based on any of the above embodiments, the predicted three-dimensional point coordinate unit is determined, and then specifically used for:

[0183] Based on the coordinates of each two-dimensional pixel and the corresponding predicted three-dimensional point, pose estimation is performed to obtain the predicted six-degree-of-freedom pose of the face.

[0184] Based on any of the above embodiments, the position coordinate unit is specifically used for:

[0185] The second sample face image is input into the face segmentation model, which obtains and outputs the position coordinates corresponding to the face; the face segmentation model is built based on the ResNet model.

[0186] Figure 7 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 7As shown, the electronic device may include: a processor 710, a communication interface 720, a memory 730, and a communication bus 740, wherein the processor 710, the communication interface 720, and the memory 730 communicate with each other through the communication bus 740. The processor 710 can call logical instructions in the memory 730 to execute a three-dimensional face reconstruction and six-degree-of-freedom pose estimation method. The method includes: acquiring a face image to be reconstructed; inputting the face image to be reconstructed into a three-dimensional face reconstruction model; obtaining and outputting the three-dimensional reconstruction result of the face image to be reconstructed and the six-degree-of-freedom pose of the face in the face image to be reconstructed by the three-dimensional face reconstruction model; the three-dimensional face reconstruction model is trained based on the three-dimensional reconstruction prediction result of the sample face image and the labeled three-dimensional reconstruction result of the sample face image, and constrained by the six-degree-of-freedom pose of the sample face image based on the correlation matrix between the three-dimensional reconstruction prediction result and the two-dimensional pixels of the face portion.

[0187] Furthermore, the logical instructions in the aforementioned memory 730 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0188] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the three-dimensional face reconstruction and six-degree-of-freedom pose estimation methods provided by the above methods. The method includes: acquiring a face image to be reconstructed; inputting the face image to be reconstructed into a three-dimensional face reconstruction model; obtaining and outputting the three-dimensional reconstruction result of the face image to be reconstructed and the six-degree-of-freedom pose of the face in the face image to be reconstructed by the three-dimensional face reconstruction model; the three-dimensional face reconstruction model is trained based on the three-dimensional reconstruction prediction result of the sample face image and the labeled three-dimensional reconstruction result of the sample face image, and is obtained by constraining the six-degree-of-freedom pose of the sample face image based on the correlation matrix between the three-dimensional reconstruction prediction result and the two-dimensional pixels of the face portion.

[0189] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements the three-dimensional face reconstruction and six-degree-of-freedom pose estimation methods provided by the above methods. The method includes: acquiring a face image to be reconstructed; inputting the face image to be reconstructed into a three-dimensional face reconstruction model; obtaining and outputting the three-dimensional reconstruction result of the face image to be reconstructed and the six-degree-of-freedom pose of the face in the face image to be reconstructed by the three-dimensional face reconstruction model; the three-dimensional face reconstruction model is trained based on the three-dimensional reconstruction prediction result of a sample face image and the labeled three-dimensional reconstruction result of the sample face image, and is obtained by constraining the six-degree-of-freedom pose of the sample face image based on the correlation matrix between the three-dimensional reconstruction prediction result and the two-dimensional pixels of the face portion.

[0190] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0191] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0192] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for three-dimensional face reconstruction and six degrees of freedom pose estimation, characterized in that, include: Obtain the image of the face to be reconstructed; The face image to be reconstructed is input into a three-dimensional face reconstruction model, and the three-dimensional reconstruction result of the face image to be reconstructed and the six-degree-of-freedom pose of the face in the face image to be reconstructed are obtained and output by the three-dimensional face reconstruction model. The three-dimensional face reconstruction model is trained based on the three-dimensional reconstruction prediction results of the sample face image and the labeled three-dimensional reconstruction results of the sample face image, and is obtained by constraining the six-degree-of-freedom pose of the sample face image based on the correlation matrix between the three-dimensional reconstruction prediction results and the two-dimensional pixels of the face part. The training steps for the 3D face reconstruction model include: Acquire the first sample face image, the second sample face image containing only the face, the true six-DOF pose of the face, the label 3D reconstruction result, the label association matrix, the label 3D coordinates corresponding to each point, and the initial 3D face reconstruction model; The initial 3D face reconstruction model and the actual six-DOF pose of the face are projected onto the image to obtain the face position coordinate annotation; Determine the position coordinates of the face in the second sample face image; Extract the global features of the face image of the first sample face image, and the three-dimensional features and local features of the face image of the second sample face image; Based on the three-dimensional features corresponding to the second sample face image, the three-dimensional reconstruction prediction result of the second sample face image is determined; Based on the global features and local features of the face image, the correlation matrix between the 3D reconstruction prediction result and the 2D pixels of the face portion is determined; Based on the correlation matrix and the 3D reconstruction prediction results, the predicted 3D point coordinates corresponding to each 2D pixel are obtained; Based on the 3D reconstruction prediction results and the labeled 3D reconstruction results, the face location coordinates and the location coordinates, the correlation matrix and the label correlation matrix, the predicted 3D point coordinates corresponding to each 2D pixel and the label 3D coordinates corresponding to each point, the parameters of the initial 3D face reconstruction model are iterated to obtain the 3D face reconstruction model.

2. The three-dimensional face reconstruction and six-degree-of-freedom pose estimation method according to claim 1, characterized in that, The method involves iterating the parameters of the initial 3D face reconstruction model based on the 3D reconstruction prediction results and the labeled 3D reconstruction results, the face location coordinate annotation and the location coordinates, the correlation matrix and the label correlation matrix, the predicted 3D point coordinates corresponding to each 2D pixel and the labeled 3D coordinates corresponding to each point, to obtain a 3D face reconstruction model, including: Based on the 3D reconstruction prediction results and the label 3D reconstruction results, the first loss is determined; Based on the face location coordinates and the location coordinates, a second loss is determined; Based on the correlation matrix and the label correlation matrix, a third loss is determined; Based on the predicted 3D point coordinates corresponding to each 2D pixel and the label 3D coordinates corresponding to each point, the fourth loss is determined. Based on the first loss, the second loss, the third loss, and the fourth loss, the parameters of the initial 3D face reconstruction model are iterated to obtain the 3D face reconstruction model.

3. The three-dimensional face reconstruction and six-degree-of-freedom pose estimation method according to claim 1, characterized in that, The step of determining the 3D reconstruction prediction result of the second sample face image based on the 3D features corresponding to the second sample face image includes: Based on the aforementioned three-dimensional features, the UV location map of the face is determined; The UV location map of the face is sequentially subjected to UV pairing and mesh sampling to obtain the three-dimensional reconstruction prediction result.

4. The three-dimensional face reconstruction and six-degree-of-freedom pose estimation method according to claim 1, characterized in that, The step of determining the correlation matrix between the 3D reconstruction prediction result and the 2D pixels of the face portion based on the global and local features of the face image includes: Two-dimensional point coordinates of the face are sampled from the position coordinates of the face in the second sample face image; Based on the mapping from the second sample face image to the first sample face image, the coordinates of all two-dimensional points in the image are obtained. Two-dimensional local features are determined based on the two-dimensional point coordinates of the face, and two-dimensional global features are determined based on the two-dimensional point coordinates of the whole image; Based on the 3D reconstruction prediction results, 3D local features are determined, and based on the 3D local features and the multilayer perceptron, 3D global features are obtained. Based on the Transformer model, the two-dimensional local features, the two-dimensional global features, the three-dimensional local features, and the three-dimensional global features are fused to obtain the correlation matrix between the three-dimensional reconstruction prediction result and the two-dimensional pixels of the face.

5. The three-dimensional face reconstruction and six-degree-of-freedom pose estimation method according to claim 1, characterized in that, Based on the correlation matrix and the 3D reconstruction prediction results, the predicted 3D point coordinates corresponding to each 2D pixel are obtained, and the process further includes: Based on the coordinates of each two-dimensional pixel and the corresponding predicted three-dimensional point, pose estimation is performed to obtain the predicted six-degree-of-freedom pose of the face.

6. The three-dimensional face reconstruction and six-degree-of-freedom pose estimation method according to claim 1, characterized in that, Determining the position coordinates of the face in the second sample face image includes: The second sample face image is input into the face segmentation model, which obtains and outputs the position coordinates corresponding to the face; the face segmentation model is built based on the ResNet model.

7. A three-dimensional face reconstruction and six-degree-of-freedom pose estimation device, characterized in that, include: The acquisition unit is used to acquire the face image to be reconstructed. The three-dimensional reconstruction unit is used to input the face image to be reconstructed into the three-dimensional face reconstruction model, and to obtain and output the three-dimensional reconstruction result of the face image to be reconstructed and the six-degree-of-freedom pose of the face in the face image to be reconstructed by the three-dimensional face reconstruction model. The three-dimensional face reconstruction model is trained based on the three-dimensional reconstruction prediction results of the sample face image and the labeled three-dimensional reconstruction results of the sample face image, and is obtained by constraining the six-degree-of-freedom pose of the sample face image based on the correlation matrix between the three-dimensional reconstruction prediction results and the two-dimensional pixels of the face part. It also includes a training unit, which is specifically used for: Acquire the first sample face image, the second sample face image containing only the face, the true six-DOF pose of the face, the label 3D reconstruction result, the label association matrix, the label 3D coordinates corresponding to each point, and the initial 3D face reconstruction model; The initial 3D face reconstruction model and the actual six-DOF pose of the face are projected onto the image to obtain the face position coordinate annotation; Determine the position coordinates of the face in the second sample face image; Extract the global features of the face image of the first sample face image, and the three-dimensional features and local features of the face image of the second sample face image; Based on the three-dimensional features corresponding to the second sample face image, the three-dimensional reconstruction prediction result of the second sample face image is determined; Based on the global features and local features of the face image, the correlation matrix between the 3D reconstruction prediction result and the 2D pixels of the face portion is determined; Based on the correlation matrix and the 3D reconstruction prediction results, the predicted 3D point coordinates corresponding to each 2D pixel are obtained; Based on the 3D reconstruction prediction results and the labeled 3D reconstruction results, the face location coordinates and the location coordinates, the correlation matrix and the label correlation matrix, the predicted 3D point coordinates corresponding to each 2D pixel and the label 3D coordinates corresponding to each point, the parameters of the initial 3D face reconstruction model are iterated to obtain the 3D face reconstruction model.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the three-dimensional face reconstruction and six-degree-of-freedom pose estimation method as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the three-dimensional face reconstruction and six-degree-of-freedom pose estimation method as described in any one of claims 1 to 6.

Images