A single image 3D face reconstruction method based on self-aligned dual regression

Abstract

A single-image face 3D reconstruction method based on self-aligned dual regression includes a training data processing stage, a network configuration stage, a training stage and a testing stage. The method of the invention enhances the unoccluded area of ​​the face based on the attention mechanism, thereby improving the robustness to occlusion; decouples the posture and shape of the 3D face, and separately learns the regression of the 3D face shape that is independent of the posture, thereby improving the accuracy of face reconstruction and weakening the negative impact of posture changes; the posture information is encoded in a rough face model as an intermediate result, and the face posture is extracted through a self-aligned post-processing process combined with the estimated face area visibility. Compared with the direct posture regression method in the existing single-image face 3D reconstruction method, the present invention has better robustness to the situation where there is occlusion and posture changes in the face image.

Description

Technical Field

[0001] The present invention belongs to the field of computer software technology, and relates to a three-dimensional face reconstruction technology, and specifically is a single-image three-dimensional face reconstruction method based on self-aligned dual regression. Background Art

[0002] Single-image 3D face reconstruction refers to restoring the 3D geometric structure of a face from a single 2D face image. It is widely used in face recognition, face editing, human-computer interaction, etc. The existing mainstream method is to reconstruct faces based on deep neural networks (DNN), which can be roughly divided into parameterized model-based methods and model-free methods.

[0003] Traditional methods based on parameterized models usually use 3DMM (3D Morphable Model), which is learned from a set of aligned 3D face models and represented by basis functions obtained by PCA (Principal Component Analysis). The face shape is reconstructed by predicting the coefficients of the PCA basis, and the face posture is obtained by predicting the rotation angle, translation, and scale of the face model separately. 3DMM-based methods can easily obtain reconstruction results with physiological rationality and complete 3D models, but such methods are highly dependent on the 3DMM used. The reconstruction accuracy is limited by the linear basis, and the reconstructed face is not discriminable enough. In recent years, some studies have improved the representation ability of the model by learning nonlinear 3DMM. For example, Luan Tran et al. used DNN as a 3DMM decoder, modeled projection, illumination, and albedo, and trained it in combination with a differentiable rendering layer, and achieved good results.

[0004] Model-free methods do not use parametric models, but directly regress the 3D geometric structure of the face. This type of method does not rely on parametric models and theoretically has a stronger ability to represent facial details. ASJackson et al. proposed VRNet, which represents the 3D face model with voxels and uses a 3D convolutional network to directly regress the face voxels. Their method does not rely on 3DMM, but has a high spatial complexity. Y.Feng et al. proposed PRNet (Joint 3D Face Reconstruction and Dense Alignment with Position Map Regression Network, Computer Vision–ECCV 2018pp 557-574), which represents the 3D face shape with a 2D UV map (UV positionmap), so that a 2D fully convolutional network can be used for regression, which significantly reduces the spatial complexity and has good reconstruction accuracy and speed.

[0005] Existing methods often perform well in controlled scenarios, but for uncontrolled scenarios, such as when the face is partially occluded or the face has a large posture deflection, the performance will be significantly reduced. The occluded area in the face image often acts as a noise signal to cause interference, and the 3D geometric structure of the occluded area lacks sufficient information for reconstruction; the 3D structure of the face has a huge solution space due to the simultaneous changes in posture and shape, and is difficult to learn. The present invention models occlusion and posture, introduces prior information, and decomposes the 3D face reconstruction problem into several more solvable sub-problems, thereby improving the robustness of the face reconstruction method to occlusion and posture changes at a lower computational cost. Summary of the invention

[0006] The problem to be solved by the present invention is that the existing single-image 3D face reconstruction method has insufficient robustness in the reconstruction when the face is partially occluded or the facial posture changes, thereby affecting the reconstruction performance and accuracy.

[0007] The technical solution of the present invention is: a single image face 3D reconstruction method based on self-aligned dual regression, which enhances the unoccluded area of ​​the face based on the attention mechanism, decouples the posture and shape of the 3D face and regresses them separately, and combines them together through a self-aligned post-processing process to complete the 3D face reconstruction, including a training data processing stage, a network configuration stage, a training stage and a testing stage:

[0008] 1) Processing training samples: Each original training sample contains a face image I in an unconstrained scene and a corresponding 3D face mesh model G. The coordinate values ​​of each vertex in G are related to the posture of the face, and G is called a posture-related face. First, G is standardized to a frontal face with the same scale, orientation, and center position, which is called a posture-independent face S. The mean of the posture-independent faces S of all samples is taken as the average face. Pose-independent face S and average face The difference is the face shape D, G, S, D is converted to UV map representation, the posture-related face G is projected onto the face image plane and simulated occlusion is added to obtain the face mask A. The foreground in the face mask A corresponds to the unoccluded area of ​​the face in the face image, and the background represents the area that does not belong to the face or the area where the face is occluded. After processing, the average face As prior knowledge, each processed training example contains a face image I as input and the expected predicted outputs A, D, S, G;

[0009] 2) Network configuration stage: Use the autoencoder based on the residual network ResNet as the basic network structure to establish a 3D face reconstruction model. The model takes the face image I as input, predicts and outputs the intermediate results A and D, and finally outputs G. The model includes the following configurations:

[0010] 2.1) Extract low-level features of the image: Use 1 convolutional layer and 6 consecutive residual blocks to extract low-level features F from the input face image I l ;

[0011] 2.2) Attention branch: The low-level features F l Input a branch network consisting of 5 convolutional layers to obtain the predicted face mask A;

[0012] 2.3) Extract high-level features of the image: perform e-exponential operation on the face mask A and the low-level features F l Point multiplication is performed to enhance the features of the unoccluded area of ​​the face, and then the high-level features F are obtained by processing with 4 consecutive residual blocks. h ;

[0013] 2.4) Dual regression module: high-level features F h Ten consecutive deconvolution layers are input, and then two groups of 7 deconvolution layers are used to obtain the face deformation D represented by UV map and the preliminary posture-related face P. The face deformation D is combined with the prior knowledge of the training sample to obtain the average face Add together to get the predicted pose-independent face S, where P contains the pose information required for face reconstruction and S contains the geometric shape information required for face reconstruction;

[0014] 2.5) Self-alignment module: Project the preliminary pose-related face P onto the image plane, combine it with the face mask A, obtain the visibility V of each point in P, use the visibility of the corresponding key points in P and S as weights, calculate the weighted similarity transformation matrix M between the corresponding key points, use M to transform S to align it to P, and obtain the pose-related face G as the final output of the reconstruction model;

[0015] 3) Training phase: The configured model is trained using training data, and the weighted mean square error (MSE) is used as the loss function for the pose-related face G, P and face deformation D. The binary cross entropy (BCE) is used as the loss function for the face mask. The Adam optimizer is used to update the parameters, and the warm up strategy is used to adjust the learning rate.

[0016] 4) Testing phase: Input the face image sequence of the test data into the trained model to obtain the reconstruction result.

[0017] Compared with the prior art, the present invention has the following advantages

[0018] The present invention proposes a method for obtaining a final face model by regressing a pose-independent face and a pose-dependent face and aligning the two. This processing method can reconstruct a more detailed face shape than a method of directly regressing a pose-dependent face; compared with a method of directly regressing a face pose and a pose-independent face, such as a method based on 3DMM, the present invention is more robust to pose changes.

[0019] The present invention adopts an occlusion-aware attention mechanism, predicts a face mask through an attention branch in a model, and the foreground of the mask is the unobstructed area of ​​the face, which can estimate the visible area of ​​the face, thereby reducing the weight of the features extracted from the occluded area. In the post-processing process, in the calculation of the weighted similarity transformation matrix of the self-alignment module, the key points use visibility as weights, the visibility of the unobstructed face area is high, and the visibility of the occluded area is low, so that the reconstruction process relies more on the unobstructed face area that is easier to accurately regress, making the face reconstruction more robust to different facial occlusion situations.

[0020] Most existing technologies cannot meet the requirements in terms of real-time performance or cannot meet the requirements of end-to-end training. The present invention proposes an efficient face reconstruction method. Through the constructed reconstruction model, real-time three-dimensional face reconstruction can be performed on the image, and end-to-end training can be performed. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 It is a system framework diagram used in the present invention.

[0022] Figure 2 It is a quantitative test result diagram of the present invention.

[0023] Figure 3 It is a sample effect diagram of the present invention.

[0024] Figure 4 It is a schematic diagram of adding simulated occlusion according to the present invention. DETAILED DESCRIPTION

[0025] The present invention proposes a single-image face 3D reconstruction method based on self-aligned dual regression, which performs occlusion detection on face images based on an attention mechanism, enhances the features of unoccluded areas, and regresses posture-related faces and posture-independent faces as intermediate results, and obtains the final 3D face reconstruction result through a self-alignment process that takes into account the visible area of ​​the face. The steps of face 3D reconstruction include processing training data, network configuration, training, and testing, which are described in detail as follows.

[0026] 1) Processing training data: Each original training example is a face image in an unconstrained scene and the corresponding face mesh model label. The face image is cropped according to the center of the face target box and scaled to 256×256×3. The correctly labeled GT (Ground Truth) face model is converted into a face deformation and posture-related face represented by a UV map, and the GT face mask is obtained based on the projection of the posture-related face on the image plane. Random noise of 0-0.2 times the size of the original target box is added to the size and center position of the face target box, and standardized to obtain a face image I and the corresponding posture-related face G. G is the format of the mesh model, that is, the face mesh model. For each posture-related face G in the training example, it is standardized to a frontal face with the same scale, orientation, and center position, which is called a posture-independent face S. The average value of the posture-independent face S of all examples is taken to obtain the average face Find the pose-independent face S and the average face The difference between G, S, D is converted to UVmap representation, the posture-related face G is projected onto the face image plane and simulated occlusion is added to obtain the face mask A, as follows:

[0027] 1. The present invention assumes that the projection of the posture-related face G onto the two-dimensional image plane is a weak perspective projection:

[0028]

[0029] Pr is the geometric structure of the face model projected onto the two-dimensional image. After projection, the value of the covered area is set to 1, and the uncovered value is set to 0. Then it is scaled to obtain a 32×32×1 face mask A.

[0030] 2. The transformation relationship between the posture-dependent face G and the posture-independent face S is:

[0031] G=f*R*S+t,

[0032] Where f is the scaling factor, R is the rotation matrix representing the face orientation, and t is the face translation. The frontal face in the pose-independent face refers to the face with the center coordinates (0,0,0) and the face orientation perpendicular to the camera plane.

[0033] 3. Posture-independent face S, average face The relationship between and face deformation D is:

[0034]

[0035]

[0036] Where N is the number of training examples.

[0037] 4. Set G, S, The way to convert D to UV map representation is to first Compute the mapping θ:p→(u,v):

[0038]

[0039] u=α1·Y+β1,

[0040]

[0041] in represent The three-dimensional coordinates of the point with the subscript p in the figure, (u, v) represents its coordinates mapped to the UV space, and α1, α2, β1, β2 are scaling and translation constants. D can obtain a 256×256×3 UV map by mapping θ, which satisfies the following properties:

[0042]

[0043] Where * refers to G, S, D, represents the UV map data of the corresponding item.

[0044] 5. Randomly add simulated occlusion patterns to the face image, and perform random rotation, symmetry, channel transformation and other data enhancement operations on the image after occlusion. The occlusion area is 0-0.3 times the area of ​​the face target frame. When occluding, set the value of the corresponding area in the corresponding face mask to 0, such as Figure 4 shown.

[0045] After processing the training samples, the training samples in the form of tensors are obtained, and the training samples (I, G, S, D, A) of the present invention are obtained from the original training samples (I, G), which are randomly loaded according to the batch size during training.

[0046] 2) Network configuration stage, i.e., face reconstruction model construction stage. Unless otherwise specified, the following convolutional layers are by default composed of convolution operation, batch normalization BN (Batch Normalization), ReLU activation function, convolution kernel is 4×4, and parameters are randomly initialized. The residual block is the Bottleneck Block structure of ResNet, which consists of 3 convolutional layers with convolution kernels of 1×1, 4×4, and 1×1, and skip connections. The details are as follows:

[0047] 2.1) Extracting low-level features of images: For each batch of face images I input into the network, the number of feature channels is first expanded to 16 by a convolutional layer, and then passed through 6 consecutive residual blocks, where the 2nd, 4th, and 6th residual blocks downsample the feature map with a step size of 2 and expand the feature channels by 1 times to obtain the low-level features F l ∈R B×32×32×128 , where B is the batch size; the specific calculation is as follows:

[0048] F l =Resblock 6 (Conv(I)), F l ∈R B×32×32×128 ,

[0049] Where conv represents the convolutional layer and Resblock represents the residual block;

[0050] 2.2) Attention branch: The low-level features F l Input a branch network consisting of 5 convolutional layers, where the 5th convolutional layer reduces the number of feature channels to 1 to obtain the output face mask A. The specific calculation is as follows:

[0051] A=Conv 5 (F l ), A∈R B×32×32×1 ;

[0052] 2.3) Extract high-level features of the image: perform e-exponential operation on the face mask A and the low-level features F l The dot product is then processed by four consecutive residual blocks, where the second and fourth residual blocks downsample the feature map with a step size of 2 and expand the feature channel by 1 to obtain the high-level feature F h ∈R B×8×8×512 , the specific calculation is as follows:

[0053]

[0054] 2.4) Dual regression module: high-level features F h Input 10 consecutive deconvolution layers, where the 4th, 7th, and 10th layers perform upsampling with a step size of 2 and reduce the feature channels to half of the previous ones, and then are processed by two sets of parallel 7-layer deconvolution layers, where the 2nd and 4th layers perform upsampling with a step size of 2 and reduce the feature channels to half of the previous ones, and the 7th layer outputs 3 channels and uses Sigmoid as the activation function. The two sets of parallel deconvolution layers obtain the preliminary pose-related face P∈R represented by UV map respectively. B×256×256×3 and face deformation D∈R B×256×256×3 , the initial pose-related face P is rougher in shape and is not the final output of the model. The pose-independent face S is obtained by adding them together. The specific calculation is as follows:

[0055] F share =Transconv 10 (F h ),F h ∈R B×8×8×512 ,

[0056] P = Transconv 7 (F share ),P∈R B×256×256×3 ,

[0057] D = Transconv 7 (F share ),D∈R B×256×256×3 ,

[0058]

[0059] Where Transconv represents the deconvolution layer;

[0060] 2.5) Self-alignment module: Calculate the normal vector of the surface where each point in P is located. Combined with the face mask A, the visibility Vis of each point in P can be obtained. The k groups of key points K in P and S that correspond to each other can be used to obtain the visibility Vis of each point in P. P and K s , with K P The visibility is used as the weight to calculate K S and K P The weighted similarity transformation M between them includes the rotation matrix R, the translation t and the scale factor f. is a 4×4 matrix, where R is a 3×3 matrix and t is 3×1. Use M to transform S to align it to P, and get the final output 3D face, that is, the posture-related face G. The specific calculation process is as follows:

[0061]

[0062]

[0063] Where i represents the i-th key point, and its three-dimensional coordinates are (x i ,y i ,z i ), the normal vector of the plane is W is a k×k weight matrix;

[0064] Calculate the mean M of the two sets of key points S and M P :

[0065]

[0066]

[0067] Where K S (i) represents the i-th key point of S, K P (i) represents the i-th key point of P, ‖W‖1 is the 1-norm;

[0068] The scaling factor in the similarity transformation is:

[0069]

[0070] The two sets of key points after normalization are:

[0071] K′ S =f*(K S -M S ),

[0072] K′ P =K P -M P ,

[0073]

[0074] U,∑,P=SVD(H),

[0075] Where SVD is the singular value decomposition of the covariance matrix H;

[0076] R=P*U T ,

[0077] t=M P -R*M S ,

[0078] R is the rotation matrix of the similarity transformation to be calculated, and t is the translation;

[0079] G=f*R*S+t,G∈R B×256×256×3 ,

[0080] G is the final output reconstructed face.

[0081] The data flow of the face reconstruction model is shown as follows Figure 1 As shown in the figure, the input face image undergoes a series of feature extraction, attention branching, dual regression, and self-alignment processing, and finally outputs a reconstructed 3D model. The input data of the model includes the face image I and the prior knowledge average face Output the posture-related face G, that is, the three-dimensional face mesh model, to achieve three-dimensional reconstruction of the image.

[0082] 3) Training phase: Use the training examples (I, G, S, D, A) processed in step 1) and prior knowledge to train the 3D face reconstruction model established in step 2), use MSE (Mean Square Error) as the loss function for the pose-related face G, P and face deformation D, use BCE (Binary Cross Entropy) as the loss function for the face mask, use Adam optimizer to update parameters, and stop after a specified number of iterations or when the model fully converges.

[0083] 4) Testing phase

[0084] The trained network parameters are loaded into the network model configured above to obtain the constructed 3D face reconstruction model. Face target detection is performed on the test data image, which is center cropped and deformed to 256×256×3 according to the detection frame and standardized. Then it is input into the 3D face reconstruction model and the 3D face reconstruction result is output.

[0085] The present invention proposes a single image face 3D reconstruction method based on self-aligned dual regression. The method is further described below through specific embodiments. After training on the 300W LP dataset and testing on the AFLW2000 dataset, the method achieves high accuracy and robustness. The method is implemented using Python 3.6 programming language and Pytorch 1.2 or higher deep learning framework.

[0086] Figure 1 This is the system framework diagram used in the present invention, and the specific implementation steps are as follows:

[0087] 1) In the data preparation stage, both training data and test data are processed in basically the same way. The face image is cropped and scaled to 256×256×3 according to the face target box, and the GT (Ground Truth) face model is converted into a face deformation and posture-related face represented by a UV map, and the GT face mask is obtained based on the projection of the posture-related face on the image plane. The difference between the processing of training and test data is that noise is added to the face detection box of the training data, and random simulated occlusion is added to the face image and face mask. Pickle is used to group and package the processed data set of 122,450 samples into 630 files to reduce the number of small file reads and speed up IO when memory is limited. After reading the image, the training program performs random data augmentation operations such as rotation, symmetry, and channel transformation, and then normalizes it. Finally, it is converted into a tensor form and randomly loaded according to the batch size.

[0088] 2) In the configuration stage of the model, the low-level features of the image are first extracted through a low-level encoder consisting of a single convolutional layer and 6 residual blocks; the low-level features are then input into the attention branch network consisting of 5 convolutional layers to obtain the face mask; the face mask is subjected to an e-exponential operation and point-multiplied with the low-level features to obtain an enhanced feature map, which is then input into a high-level encoder consisting of 4 residual blocks to obtain high-level features; the high-level features are then passed through two sets of parallel 17-layer deconvolution decoders to obtain the posture-related face and posture-independent face as intermediate results. The two decoders share the first 10 layers of deconvolution layers with larger parameters, but the last 7 layers close to the output end are independent of each other, and the activation function of the last layer is Sigmoid. ; Take out two sets of key points that correspond to each other from the pose-related face and the pose-independent face of the intermediate result, calculate the visibility of the key points of the pose-related face according to the normal vector of the plane where the key points are located and the face mask, and then use the visibility as the weight to perform weighted averaging and normalization on the two sets of key points, and at the same time obtain the scale factor between the two sets of key points, and then use the visibility as the weight to calculate the weighted covariance matrix, perform SVD on the covariance matrix to calculate the optimal rotation transformation matrix between the two sets of key points after normalization, and then obtain the offset of the similarity transformation between the original key points, so far the optimal similarity transformation between the two sets of key points has been calculated; use the similarity transformation to align the pose-independent face to the pose-related face, and thus obtain the final reconstruction result. The calculation process is implemented in a differentiable way, so that the network model can be fully run on the GPU for end-to-end training.

[0089] 3) In the training phase, this embodiment uses MSE (Mean Square Error) as the loss function for posture-related faces G, P and face deformation D, uses BCE (Binary Cross Entropy) as the loss function for face masks, and takes the absolute value of the difference between each point and the mean of the points in its 4 neighborhoods as the smoothness loss function. The Adam optimizer is used for parameter update. The batch size is set to 16, that is, 16 samples are taken from the training set for network training in each iteration. The learning rate is adjusted using the warm up strategy. The initial learning rate is 0.00001, which increases uniformly to 0.0001 after 4 rounds of training, and then exponentially decays with a decay rate of 0.85. Trained on 1 GTX 1080 Ti GPU.

[0090] 4) Testing phase

[0091] The test data was processed in the same way as the training data. The face target box was center-cropped and deformed to 256*256 and standardized as the input image. It was tested on the tasks of face 3D reconstruction, 2D and 3D sparse face alignment (68 points calculated in total), and 2D and 3D dense face alignment (45,000 points calculated). The test indicators used were all NME (normalized mean error). In addition, to test the performance under different facial postures, samples in the range of facial yaw angles of 0-30 degrees, 30-60 degrees, and 60-90 degrees were tested respectively. The average prediction speed of the model is 70 frames per second. The quantitative test results are shown in Figure 2 Compared with PRNet proposed by Y. Feng et al., the proposed method has relatively stable reconstruction performance under different facial postures, and the average error is at a lower level. The effect of the proposed method on the test set is as follows: Figure 3 As shown, the first row is face images, including cases where the face is partially occluded and the facial posture changes. The second row of images is the result of sparse alignment, which mainly displays the key points and contours of the reconstructed face model. The third row is the reconstructed three-dimensional model. It can be seen that for face images in unconstrained scenarios, such as partial occlusion, exaggerated facial expressions, large facial deflections, etc., the face model reconstructed by the present invention can visually fit the face in the image, which confirms the results of the quantitative test.

Claims

1. A single image face 3D reconstruction method based on self-aligned dual regression, characterized by Based on the attention mechanism, the unobstructed area of ​​the face is enhanced, the posture and shape of the 3D face are decoupled and regressed separately, and combined through the self-alignment post-processing process to complete the 3D face reconstruction, including the training data processing stage, network configuration stage, training stage and testing stage: 1) Processing training samples: Each original training sample contains a face image I in an unconstrained scene and a corresponding 3D face mesh model G. The coordinate values ​​of each vertex in G are related to the posture of the face, and G is called a posture-related face. First, G is standardized to a frontal face with the same scale, orientation, and center position, which is called a posture-independent face S. The mean of the posture-independent faces S of all samples is taken as the average face. Pose-independent face S and average face The difference is the face shape D, G, S, D is converted to UV map representation, the posture-related face G is projected onto the face image plane and simulated occlusion is added to obtain the face mask A. The foreground in the face mask A corresponds to the unoccluded area of ​​the face in the face image, and the background represents the area that does not belong to the face or the area where the face is occluded. After processing, the average face As prior knowledge, each processed training example contains a face image I as input and the expected predicted outputs A, D, S, G; 2) Network configuration stage: Use the autoencoder based on the residual network ResNet as the basic network structure to establish a 3D face reconstruction model. The model takes the face image I as input, predicts and outputs the intermediate results A and D, and finally outputs G. The model includes the following configurations: 2.1) Extract low-level features of the image: Use 1 convolutional layer and 6 consecutive residual blocks to extract low-level features F from the input face image I l ; 2.2) Attention branch: The low-level features F l Input a branch network consisting of 5 convolutional layers to obtain the predicted face mask A; 2.3) Extract high-level features of the image: perform e-exponential operation on the face mask A and the low-level features F l Point multiplication is performed to enhance the features of the unoccluded area of ​​the face, and then the high-level features F are obtained by processing with 4 consecutive residual blocks. h ; 2.4) Dual regression module: high-level features F h Ten consecutive deconvolution layers are input, and then two groups of 7 deconvolution layers are used to obtain the face deformation D represented by UV map and the preliminary posture-related face P. The face deformation D is combined with the prior knowledge of the training sample to obtain the average face Add together to get the predicted pose-independent face S, where P contains the pose information required for face reconstruction and S contains the geometric shape information required for face reconstruction; 2.5) Self-alignment module: Project the preliminary pose-related face P onto the image plane, combine it with the face mask A, obtain the visibility V of each point in P, use the visibility of the corresponding key points in P and S as weights, calculate the weighted similarity transformation matrix M between the corresponding key points, use M to transform S to align it to P, and obtain the pose-related face G as the final output of the reconstruction model; 3) Training phase: The configured model is trained using training data, and the weighted mean square error (MSE) is used as the loss function for the pose-related face G, P and face deformation D. The binary cross entropy (BCE) is used as the loss function for the face mask. The Adam optimizer is used to update the parameters, and the warm up strategy is used to adjust the learning rate. 4) Testing phase: Input the face image sequence of the test data into the trained model to obtain the reconstruction result.

2. The method for 3D reconstruction of a single-image face based on self-aligned dual regression according to claim 1, characterized in that In step 1), all samples are preprocessed, the face image is cropped and scaled to 256×256×3 according to the face target box, and the GT face model is converted into a face deformation and posture-related face represented by a UV map, and the GT face mask is obtained according to the projection of the posture-related face on the image plane. Noise is added to the face detection frame of the training data, and random simulated occlusion is added to the face image and face mask. The image after adding occlusion is subjected to random rotation, symmetry, and channel transformation data enhancement operations, and then standardized. Finally, the training samples are converted into tensor Tensor form (I, G, S, D, A) for random loading according to the batch size during training.

3. The method for 3D reconstruction of a single-image face based on self-aligned dual regression according to claim 1, characterized in that In the network configuration of step 2), the convolution layer consists of convolution operation, batch normalization BN and ReLU activation function, the convolution kernel is 4×4, the parameters are randomly initialized, and the residual block is the Bottleneck Block structure of ResNet, which consists of 3 convolution layers with convolution kernels of 1×1, 4×4, and 1×1 respectively, and skip connections.

4. The method for 3D reconstruction of a single-image face based on self-aligned dual regression according to claim 1, characterized in that In step 2.5), the visibility of a point on the face is calculated by calculating the normal vector of the plane where the point is located. If the component of the normal vector in the z-axis direction is less than 0, the visibility is 0, otherwise it is the corresponding value of the point in the face mask A after being projected onto the image plane.

5. The method for 3D reconstruction of a single-image face based on self-aligned dual regression according to claim 1, characterized in that In step 2.5), the transformation of the posture-independent face S includes scaling, rotation and translation: two groups of key points that correspond to each other are taken from the posture-independent face S and the preliminary posture-related face P, and the two groups of key points are weighted averaged and normalized with visibility as weight, and the scale scaling factor between the two groups of key points is obtained at the same time, and then the covariance matrix of the two groups of points after normalization is calculated with visibility as weight, and the covariance matrix is ​​subjected to singular value decomposition SVD to calculate the optimal rotation transformation matrix between the two groups of key points after normalization, that is, the weighted similarity transformation matrix M, and the posture-independent face S is rotated by the weighted similarity transformation matrix M, and the posture-independent face S is scaled by the scale scaling factor, and the average value of the difference between the rotation and scaling and the preliminary posture-related face P is used as the translation in the similarity transformation.

Images