A single image based fine-grained 3D face reconstruction method

Abstract

This invention relates to a fine-grained 3D face reconstruction method based on a single image. The method preprocesses the original face image dataset by cropping face regions, extracting face mask images, and labeling landmarks. Then, in a coarse 3D face reconstruction stage, the preprocessed face image is input into a face regression model for network model training. Based on the trained face regression model parameters, a face coefficient vector is output, and a smooth 2D face image is generated through rendering. In the fine-grained 3D face reconstruction stage, an albedo map is input into an albedo model for albedo network model training. Based on the trained albedo model parameters, an albedo map is output, and based on the trained depth shift model parameters, a depth shift map is output. This method can reconstruct fine-grained 3D face geometry and texture information from a single image and is robust to face images under different conditions in nature, making it applicable to virtual reality and entertainment.

Description

Technical Field

[0001] This invention relates to the field of 3D face reconstruction and graphics and image processing technology, and in particular to a fine-grained 3D face reconstruction method based on a single image. Background Technology

[0002] With the booming development of virtual reality technology and the entertainment and film industry, 3D face reconstruction has received widespread attention and has high research and practical value. Traditional 3D face reconstruction techniques, such as stereoscopic vision and laser scanning, require recovering the three-dimensional structure of a specific face from multiple face images. However, these methods are computationally complex, have low real-time performance, limited application scenarios, and require expensive equipment.

[0003] With the rapid development of deep learning, in recent years, more and more researchers have focused on 3D reconstruction based on single face images, achieving significant improvements. Compared with traditional methods, single-image-based 3D face reconstruction methods can achieve a balance between accuracy and cost, offering advantages such as high flexibility, strong adaptability, and wider application scenarios. However, because two-dimensional face images provide too little information and are easily affected by occlusion, extreme poses, or harsh lighting conditions, the 3D faces reconstructed from single images are too smooth, making it difficult to recover details such as wrinkles that change with facial expressions. Therefore, how to reconstruct high-fidelity, fine-grained 3D faces from single face images obtained from natural environments is a key problem that urgently needs to be solved. Summary of the Invention

[0004] The purpose of this invention is to address the shortcomings of existing technologies by proposing a fine-grained 3D face reconstruction method based on a single image. This method uses a coarse-to-fine structure for fine-grained 3D reconstruction of a single face image. This method can not only recover rich facial structural information such as wrinkles that change with facial expressions, but also has robustness to face images under occlusion, extreme poses, and different lighting conditions.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] A fine-grained 3D face reconstruction method based on a single image, the steps of which are as follows:

[0007] Step 1: Preprocess the original face image dataset, crop the face region, extract the face mask image and label 68 landmarks;

[0008] Step 2: In the coarse 3D face reconstruction stage, the preprocessed face image is input into the face regression model for network model training;

[0009] Step 3: Based on the face regression model parameters trained in Step 2, output the face coefficient vector, reconstruct a rough 3D face using the 3DMM (3DMorphable Model) prior model, and generate a smooth 2D face image through rendering.

[0010] Step 4: In the fine-grained 3D face reconstruction stage, the albedo map generated in Step 3 is input into the albedo model for albedo network model training; UV transformation is performed on the input face image and the smooth face image rendered in Step 3 respectively, and the transformed UV map is input into the depth shift model for depth shift network model training.

[0011] Step 5: Based on the albedo model parameters obtained after training in Step 4, output a detailed albedo map and a depth shift map based on the depth shift model parameters obtained after training in Step 4; combine the 3D face reconstructed in Step 3 with its detailed albedo map and depth shift map to synthesize a fine-grained 3D face, and generate a fine-grained face image through a differentiable renderer.

[0012] Preferably, the preprocessing in step 1 includes the following operations:

[0013] 1-1: For the original face image, use the face-alignment method to crop out the face region, and then scale the size of the cropped face image to 300×300;

[0014] 1-2: 68 landmarks were extracted using the face landmark detection method FAN;

[0015] 1-3: Obtain the face mask image using the 3DDFA_V2 method;

[0016] 1-4: The final result is 191,081 face dataset pairs, including processed face images and corresponding 68 landmarks and mask images. 181,081 face dataset pairs will be randomly selected for training, and the remaining 10,000 face images will be used for testing.

[0017] Preferably, the coarse-stage face regression model training in step 2 includes the following operations:

[0018] 2-1: Scale the face images in the training set to 224×224;

[0019] 2-2: Input the face image into the face regression model R-Net for training;

[0020] 2-3: The loss function consists of four loss terms: photometric loss, marker loss, identity perception loss, and regularization loss.

[0021] 2-4: The face regression model R-Net uses Adam's optimization algorithm with a learning rate of 0.00001, a batch size of 16, and a total of 250,000 iterations of training.

[0022] Preferably, the 3D face reconstruction and rendering results in the roughing stage of step 3 include the following operations:

[0023] 3-1: After 250,000 iterations of training, the training parameters of the corresponding face regression model R-Net are obtained;

[0024] 3-2: Input the test set into the trained face regression model R-Net to obtain the face coefficient vector predicted by the model, including the face's identity, albedo, expression, pose, and illumination;

[0025] 3-3: A coarse 3D face and albedo map are reconstructed using a 3DMM prior model based on the face's identity, albedo, and expression parameter vectors;

[0026] 3-4: Use a differentiable renderer to render and generate smooth facial images based on the pose and lighting coefficients of the face.

[0027] Preferably, the training of the albedo model and depth shift model in the fine-grained stage of step 4 includes the following operations:

[0028] 4-1: Input the smoothed albedo map generated in step 3 into the albedo model Albedo-Net for training;

[0029] 4-2: Perform UV transformation on the input face image and the smooth face image rendered in step 3 respectively. Subtract the transformed UV images at the pixel level to obtain the UV shift map. Input the UV shift map into the deep shift model Disp-Net for training.

[0030] 4-3: The loss function consists of four loss terms: photometric loss, perceptual loss, albedo loss, and smoothing loss.

[0031] 4-4: The albedo-Net and deep shift model Disp-Net were trained using Adam's optimization algorithm with a learning rate of 0.00005, a batch size of 10, and a total of 250,000 iterations.

[0032] Preferably, the fine-grained face image rendering result in step 5 includes the following operations:

[0033] 5-1: After 250,000 iterations of training, the training parameters of the corresponding albedo model Albedo-Net and the deep shift model Disp-Net were obtained respectively.

[0034] 5-2: Input the smoothed albedo map generated in step 3 into the trained albedo model Albedo-Net, and output an albedo map with detailed information;

[0035] 5-3: Perform UV transformation on the input face image and the smooth face image rendered in step 3 respectively. Subtract the transformed UV images at the pixel level to obtain the UV shift map. Input the UV shift map into the trained depth shift model Disp-Net to output a depth shift map with rich details.

[0036] 5-4: Combine the coarse 3D face reconstructed in step 3 with its albedo map and depth shift map containing detailed information to synthesize a fine-grained 3D face. Then, use a differentiable renderer to render the 3D face and generate a high-fidelity face image.

[0037] Compared with the prior art, the present invention has the following obvious and prominent substantive features and significant advantages:

[0038] 1. This invention designs a coarse-to-fine 3D face reconstruction framework. In the coarse stage, a smooth 3D face can be accurately reconstructed from a single input face image. In the fine stage, by designing the depth-shifting model Disp-Net and the albedo model Albedo-Net, the structural information of the fine-grained 3D face can be effectively recovered, and highly realistic face images can be generated through rendering.

[0039] 2. Compared with existing methods, the present invention can reconstruct high-fidelity 3D facial geometry and texture information from a single image, and is also robust to facial images under extreme conditions in nature, which is of great significance for the development of virtual reality and entertainment film and television. Attached Figure Description

[0040] Figure 1 This is a flowchart of the method of the present invention.

[0041] Figure 2 This is an overall flowchart of the method of the present invention.

[0042] Figure 3 This is the network structure of Albedo-Net, the albedo model of the method of this invention.

[0043] Figure 4 This is the network structure of Disp-Net, the deep shift model of the method of this invention.

[0044] Figure 5 The image shows the 3D face reconstruction result of a single face image using a coarse-to-fine framework designed for the method of this invention.

[0045] Figure 6 and Figure 7 The image shows a comparison of the 3D face reconstruction results from a single face image using a coarse-to-fine framework designed for the method of this invention, with the results from several existing methods. Detailed Implementation

[0046] The preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0047] Example 1:

[0048] See Figure 1 A fine-grained 3D face reconstruction method based on a single image includes the following steps:

[0049] Step 1: Preprocess the original face image dataset, crop the face region, extract the face mask image and label 68 landmarks;

[0050] Step 2: In the coarse 3D face reconstruction stage, the preprocessed face image is input into the face regression model for network model training;

[0051] Step 3: Based on the face regression model parameters trained in Step 2, output the face coefficient vector, reconstruct a rough 3D face using the 3DMM prior model, and generate a smooth 2D face image through rendering.

[0052] Step 4: In the fine-grained 3D face reconstruction stage, the albedo map generated in Step 3 is input into the albedo model for albedo network model training. UV transformation is performed on the input face image and the smoothed face image rendered in Step 3, and the transformed UV maps are input into the depth shift model for depth shift network model training.

[0053] Step 5: Based on the albedo model parameters obtained after training in Step 4, output a detailed albedo map, and based on the depth shift model parameters obtained after training in Step 4, output a depth shift map. Combine the 3D face reconstructed in Step 3 with its detailed albedo map and depth shift map to synthesize a fine-grained 3D face, and generate a fine-grained face image through a differentiable renderer.

[0054] The method in this embodiment can reconstruct 3D facial structural features with rich details and realistic texture information from a single face image. It is simple, easy to implement, and low in cost, without requiring expensive hardware or specific application scenarios.

[0055] Example 2:

[0056] This second embodiment is basically the same as the first embodiment, with the following differences:

[0057] The preprocessing in step 1 includes the following operations:

[0058] 1-1: For the original face image, use the face-alignment method to crop out the face region, and then scale the size of the cropped face image to 300×300;

[0059] 1-2: 68 landmarks were labeled using the FAN (Face Anonymous Array) method for detecting facial landmarks;

[0060] 1-3: Use the 3DDFA_V2 method to obtain the face mask image, where 1 represents the face region and 0 represents the non-face region;

[0061] 1-4: Finally, 191,081 face dataset pairs were obtained, including preprocessed face images and corresponding 68 landmarks and mask images. 181,081 face dataset pairs were randomly selected for training, and the remaining 10,000 face images were used for testing.

[0062] The training of the face regression network model in step 2 includes the following operations:

[0063] 2-1: Scale the face images in the training set to 224×224;

[0064] 2-2: Input the face image as follows Figure 2 The face regression model R-Net based on the face recognition network VGG-Face is used for training. In this invention, a fully connected layer is used to replace the last network layer of VGG-Face to regress the low-dimensional face parameter vector.

[0065] 2-3: The loss function consists of four loss terms: photometric loss, marker loss, identity perception loss, and regularization loss.

[0066] The expression for the loss function is as follows:

[0067] L coarse =λ pho L pho +λ lmk L lmk +λid L id +λ reg L reg (1)

[0068] Among them, L coarse L represents the loss function for the coarse 3D face reconstruction stage. pho L represents luminous loss. lmk It is the marker loss, L id It is perceived loss, L reg For regularization loss, λ pho , λ lmk , λ id and L reg These are the weight coefficients corresponding to photometric loss, marker loss, identity perception loss, and regularization loss, respectively.

[0069] To improve the accuracy of 3D face reconstruction, this invention uses the photometric loss function L. pho Enhance the similarity between the rendered face image in step 3 and the input face image. pho The detailed expression is as follows:

[0070]

[0071] In the formula, M represents the index of the image pixels, and I represents the input face image. R V represents the smooth face image rendered in step 3. V is the face mask image obtained by the 3DDFA_V2 method in step 1, where 1 represents the face region and 0 represents the non-face region.

[0072] To effectively align the pose and expression of the reconstructed 3D face, this invention uses a marker-based loss function L. lmk To reduce the error between the 68 landmarks of the input face image and the 68 landmarks of the rendered face image. lmk The detailed expression is as follows:

[0073]

[0074] Where, q i This represents the i-th landmark point in the input image I. This refers to the face image I rendered in step 3. R The i-th landmark is N=68, which represents the number of face landmarks.

[0075] To improve the accuracy of reconstructing 3D faces with specific identities, this invention uses the identity-aware loss function L. idTo reduce the identity error between the input face image and the face image rendered in step 3. id The detailed expression is as follows:

[0076]

[0077] In the formula, φ(I) and φ(I) R The ) represents the feature vectors extracted by the pre-trained face recognition model VGG-Face from the input face image and the face image rendered in step 3, respectively.

[0078] Furthermore, to prevent distortion in the reconstructed 3D face, this invention uses a regularized loss function L. reg To prevent the values ​​of the face identity, expression, and albedo coefficient vectors predicted by the face regression model R-Net in step 3 from being too large. reg The detailed expression is as follows:

[0079] L reg =λ α ||α|| 2 +λ β ||β|| 2 +λ δ ||δ|| 2 (5)

[0080] Where α, β, and δ represent the face identity, expression, and albedo coefficient vectors predicted by the face regression model R-Net in step 3, respectively, and λ α , λ β and λ δ These represent the weight parameters corresponding to these regularization terms.

[0081] 2-4: The face regression model R-Net uses Adam's optimization algorithm with a learning rate of 0.00001, a batch size of 16, and a total of 250,000 iterations of training.

[0082] The roughing-out stage of 3D face reconstruction and rendering in step 3 includes the following operations:

[0083] 3-1: After 250,000 iterations of training, the training parameters of the corresponding face regression model R-Net are obtained;

[0084] 3-2: Input the face images I from the test set into the trained face regression model R-Net to obtain the face coefficient vector predicted by the model. Among them, the identity coefficient of facial recognition albedo Facial expression coefficient posture and illumination coefficient

[0085] 3-3: A coarse 3D face shape and albedo map are reconstructed using the 3DMM prior model based on the face's identity α, albedo β, and expression coefficient vector δ. The detailed expressions for the 3D face shape and albedo are as follows:

[0086]

[0087]

[0088] In the formula and These represent the average values ​​of 3D face shape and albedo, respectively. and Let α, β, and δ represent the basis of the principal components of face identity, expression, and albedo, respectively, and let α, β, and δ represent the coefficient vectors of face identity, expression, and albedo, respectively. The basis B of the principal components of face identity used in this invention... id and the principal component of albedo B t Based on the Bézier face model (BFM), the base B of the main components of facial expression. exp Derived from the Facewarehouse model. Ultimately, the expression for the 3D face is as follows:

[0089]

[0090] Where N = 53215 represents the number of vertices contained in a 3D face.

[0091] 3-4: A differentiable renderer is used to render and generate a smooth face image based on the face pose and lighting coefficient vectors. The expression for the camera model is as follows:

[0092] Because the training set uses natural human face images, this invention employs a perceptual camera model to project 3D faces onto a two-dimensional image space based on face pose coefficient vectors. The face pose coefficient vector is defined as p = {R, T}, where R represents the rotation vector and T represents the translation vector. The expression for projecting a 3D face onto a two-dimensional image space is as follows:

[0093] v = Π(fRM) i +t) (9)

[0094] In the formula M i ∈R 3 Represents the vertex position vector of a 3D face, f represents the scaling factor of the camera model, Π∈R 2×3 This represents an orthogonal transformation from three dimensions to two dimensions, where v represents a two-dimensional vertex vector.

[0095] This invention uses spherical harmonics (SH) to simulate the ambient lighting model; therefore, the expression for the surface shading of the human face is as follows:

[0096]

[0097] Where b=3 represents the spherical harmonic band, γ represents the face illumination coefficient vector, and t i Skin texture representing albedo T, n i φ represents the surface normal vector of a 3D face. b ∈R 3 →R represents the spherical harmonic basis function, and e represents the Adama product.

[0098] Finally, the identity coefficient of the face is obtained through step 3-2. albedo Facial expression coefficient posture and illumination coefficient The method of this invention can use a differentiable renderer to render and generate a smooth face image from the rough 3D face reconstructed in step 3-3. The expression for face image rendering is as follows:

[0099] I R =R(M(α,β,δ),p,C(γ)) (11)

[0100] In the formula, R represents the rendering operation.

[0101] The training of the albedo model and depth shift model in the fine-grained 3D face reconstruction stage in step 4 includes the following operations:

[0102] 4-1: Input the smoothed albedo map generated in step 3 into the albedo model Albedo-Net for training. The complete network structure of the albedo model Albedo-Net is as follows: Figure 3 As shown:

[0103] This invention uses the U-shaped albedo model Albedo-Net, which includes 10 downsampling layers and 10 upsampling layers. The network input is a smooth albedo map with 3 channels and a size of 512×512 obtained from step 3. The output is an albedo map with 3 channels and a size of 512×512 containing detailed information. The first downsampling convolutional layer uses a 3×3 kernel with a stride of 1 and employs the LeRU activation function, maintaining the same output feature map size with 16 channels. The subsequent 9 downsampling layers use 3×3 kernels with a stride of 2 and employ the LeRU activation function, decreasing the size of the output feature map by a factor of 2 each time. The first 9 upsampling deconvolutional layers use 3×3 kernels with a stride of 2, employ the LeRU activation function, and use skip connections that sum the feature vectors, increasing the size of the output feature map by a factor of 2 each time, ultimately resulting in a 512×512 output feature map. The final upsampling deconvolutional layer uses a 3×3 kernel with a stride of 1 and employs a sigmoid activation function to produce a final albedo map of size 512×512 with detailed information. This invention trains the albedo model Albedo-Net in an unsupervised manner, without using a ground-truth dataset.

[0104] 4-2: Compare the input face image I and the smoothed face image I rendered in step 3. R UV transformations are performed separately, and the transformed UV images are subtracted pixel-wise to obtain a UV-shifted image. This UV-shifted image is then input into the deep shift model Disp-Net for training. The complete network structure of the deep shift model Disp-Net is as follows: Figure 4 As shown:

[0105] This invention uses the same structure as the depth-shifting model Disp-Net and the albedo model Albedo-Net, each containing 10 downsampling layers and 10 upsampling layers. The network input is a 512×512 UV-shifted map with 3 channels, and the output is a 512×512 depth-shifted map with rich facial structural information, containing 1 channel. The first downsampling convolutional layer uses a 3×3 kernel with a stride of 1 and employs the LeRU activation function, maintaining the same output feature map size with 16 channels. The subsequent 9 downsampling layers use 3×3 kernels with a stride of 2 and employ the LeRU activation function, decreasing the output feature map size by a factor of 2 each time. The first 9 upsampling deconvolutional layers use 3×3 kernels with a stride of 2, employ the LeRU activation function, and use skip connections that sum feature vectors, increasing the output feature map size by a factor of 2 each time, resulting in a final output feature map size of 512×512. The final upsampling deconvolutional layer uses a 3×3 kernel with a stride of 1 and employs the Tanh activation function to produce a 512×512 depth shift map with rich information such as wrinkles that change with facial expressions. This invention trains the Disp-Net depth shift model in an unsupervised manner without using the ground-truth dataset.

[0106] 4-3: The loss function consists of four loss terms: photometric loss, identity perception loss, albedo loss, and smoothing loss.

[0107] The expression for the loss function is as follows:

[0108] L fine =λ pho L pho +λ id L id +λ alb L alb +λ smo L smo (12)

[0109] Among them, L fine L represents the loss function for the fine-grained 3D face reconstruction stage. pho and L id L represents the photometric loss and identity perception loss in the coarse 3D face reconstruction stage of steps 2-3, respectively. alb It is albedo loss, L smo To smooth the loss, λ pho , λ id , λ alb and L smoThese are the weight coefficients corresponding to photometric loss, identity perception loss, albedo loss, and smoothing loss, respectively.

[0110] This invention uses the albedo loss function L alb It can effectively prevent artifacts from appearing in the generated albedo map. alb The detailed expression is as follows:

[0111]

[0112] In the formula A represents the detailed albedo map output by the model, and A represents the smooth albedo map generated in step 3.

[0113] Furthermore, in order to effectively reduce the distortion caused by facial images under conditions such as occlusion and large poses on the reconstructed 3D face, this invention uses a smoothing loss function L. smo To improve the similarity between adjacent pixel values ​​in the normal vector map and the depth shift map, L smo The detailed expression is as follows:

[0114]

[0115] Where Vn(i) represents the i-th pixel difference between the normal vector map generated in the coarse stage and the normal vector map generated in the fine-grained stage, and Vd(i) represents the i-th pixel difference between the depth map generated in the coarse stage and the depth map generated in the fine-grained stage. uv Let N(i) represent a vertex in the UV space, and let N(i) represent the set of adjacent vertices with a radius of 1 from the i-th vertex. n and w d These are the weighting coefficients for the two loss terms.

[0116] 4-4: The albedo-Net and deep shift model Disp-Net were trained using Adam's optimization algorithm with a learning rate of 0.00005, a batch size of 10, and a total of 250,000 iterations.

[0117] The fine-grained 3D face reconstruction and rendering in step 5 includes the following operations:

[0118] 5-1: After 250,000 iterations of training, the training parameters of the corresponding albedo model Albedo-Net and deep shift model Disp-Net are obtained;

[0119] 5-2: Input the smoothed albedo map generated in step 3 into the trained albedo model Albedo-Net, and output an albedo map with detailed information;

[0120] 5-3: Combine the input face image I with the smoothed face image I rendered in step 3. R UV transformation is performed separately, and the transformed UV maps are subtracted pixel by pixel to obtain a UV-shifted map. The UV-shifted map is then input into the trained depth shifting model Disp-Net to output a depth shifted map with rich detail information.

[0121] 5-4: Combine the coarse 3D face reconstructed in step 3 with its albedo map and depth shift map, which contain rich facial details, to synthesize a fine-grained 3D face. Then, use a differentiable renderer to render the 3D face and generate a highly realistic face image.

[0122] This embodiment selects a subset of images from the public face image datasets CelebA and LFW to verify the performance of the method. The method is used to perform fine-grained 3D face reconstruction on face images, and compared with state-of-the-art coarse 3D face reconstruction methods: RingNet, Deep3DFace, 3DDFA_V2, MGCNet, and fine-grained 3D face reconstruction methods: Nonlinear_Face, Facescape, Unsupervised-detail, and DECA. The reconstruction results of the method are as follows: Figure 5 As shown, the normal vector map, face geometry, and rendering results of the 3D face reconstructed in the fine-grained stage have significantly richer details in facial structure and texture information than the 3D face reconstructed in the coarse-grained stage, and the realism is also higher. Figure 6 and Figure 7 This section compares the 3D face geometry and rendering results reconstructed using the method of this invention with those of state-of-the-art existing methods. Figure 6 The comparison results show that the method of this invention can reconstruct more accurate and fine-grained facial geometry from facial images with different poses and occlusions, and recover richer information such as wrinkles that change with facial expressions. Furthermore, the method... Figure 7 It is evident that the method of the present invention produces more realistic 3D face rendering effects and richer texture details under different lighting, poses, and occlusion conditions.

[0123] The embodiments of the present invention have been described above in conjunction with the accompanying drawings. However, the present invention is not limited to the above embodiments. Various changes can be made according to the purpose of the invention. Any changes, modifications, substitutions, combinations or simplifications made based on the spirit and principle of the technical solution of the present invention shall be equivalent substitutions. As long as they meet the purpose of the invention and do not deviate from the technical principle and inventive concept of the present invention, they shall fall within the protection scope of the present invention.

Claims

1. A fine-grained 3D face reconstruction method based on a single image, characterized in that, The operation steps are as follows: Step 1: Preprocess the original face image dataset, crop the face region, extract the face mask image, and label the face region with 68 landmarks; Step 2: In the coarse 3D face reconstruction stage, the cropped face image is input into the face regression network for network model training; Step 3: Based on the training parameters obtained after training the face regression model, output the face coefficients, reconstruct a rough 3D face using the 3DMM prior model, and generate a smooth face image using a differentiable renderer. Step 4: In the fine-grained 3D face reconstruction stage, the albedo map generated in Step 3 is input into the albedo model for albedo network model training; UV transformation is performed on the input face image and the smooth face image generated in Step 3 respectively, and the transformed UV map is input into the depth shift model for depth shift network model training. Step 5: Based on the training parameters obtained after training the albedo model, output a detailed albedo map, and based on the training parameters obtained after training the depth shift model, output a depth shift map; combine the 3D face reconstructed in Step 3 with its detailed albedo map and depth shift map to synthesize a fine-grained 3D face, and generate a fine-grained face image through a differentiable renderer.

2. The fine-grained 3D face reconstruction method based on a single image according to claim 1, characterized in that, The preprocessing in step 1 includes the following specific steps: 1-1: For the original face image, use the face-alignment method to crop out the face region, and then scale the size of the cropped face image to 300×300; 1-2: 68 landmarks were extracted from the face image using the FAN (Face Anchor Detection) method; 1-3: Obtain the face mask image using the 3DDFA_V2 method; 1-4: Finally, 191,081 face dataset pairs were obtained, including cropped face images, landmarks, and mask images. 181,081 face dataset pairs were randomly selected for training, and the remaining 10,000 face images were used for testing.

3. The fine-grained 3D face reconstruction method based on a single image according to claim 1, characterized in that, The face regression model training in the coarse stage of step 2 includes the following specific steps: 2-1: Scale the face images in the training set to 224×224; 2-2: Input the face image into the face regression model R-Net for training; 2-3: The loss function consists of four loss terms: photometric loss, marker loss, identity perception loss, and regularization loss. 2-4: The face regression model uses Adam's optimized algorithm with a learning rate of 0.00001, a batch size of 16, and a total of 250,000 iterations of training.

4. The fine-grained 3D face reconstruction method based on a single image according to claim 1, characterized in that, The smooth face image rendering result in step 3 includes the following specific operation steps: 3-1: After 250,000 iterations of training, the training parameters of the corresponding face regression model R-Net are obtained; 3-2: Input the test set into the trained face regression model to obtain the face coefficient vector predicted by the model, including the face's identity, albedo, expression, pose, and illumination; 3-3: A coarse 3D face and albedo map are reconstructed using the identity, albedo, and expression coefficient vectors of the face through the 3DMM prior model; 3-4: Use a differentiable renderer to render 3D faces based on their pose and lighting parameters, and generate smooth face images.

5. The fine-grained 3D face reconstruction method based on a single image according to claim 1, characterized in that, The training of the albedo model and depth shift model in the fine-grained stage of step 4 includes the following specific steps: 4-1: Input the smoothed albedo map generated in step 3 into the albedo model Albedo-Net for training; 4-2: Perform UV transformation on the input face image and the smooth face image rendered in step 3 respectively. Subtract the transformed UV images at the pixel level to obtain the UV shift map. Input the UV shift map into the deep shift model Disp-Net for training. 4-3: The loss function consists of four loss terms: photometric loss, perceptual loss, albedo loss, and smoothing loss. 4-4: The albedo-Net and deep shift model Disp-Net were trained using Adam's optimization algorithm with a learning rate of 0.00005, a batch size of 10, and a total of 250,000 iterations.

6. The fine-grained 3D face reconstruction method based on a single image according to claim 1, characterized in that, The fine-grained face image rendering result in step 5 includes the following specific operation steps: 5-1: After 250,000 iterations of training, the training parameters of the corresponding albedo model Albedo-Net and deep shift model Disp-Net are obtained; 5-2: Input the smoothed albedo map generated in step 3 into the trained albedo model Albedo-Net, and output an albedo map with detailed information; 5-3: Perform UV transformation on the input face image and the smooth face image rendered in step 3 respectively. Subtract the transformed UV images at the pixel level to obtain the UV shift map. Input the UV shift map into the trained depth shift model Disp-Net to output a depth shift map with rich details. 5-4: Combine the coarse 3D face reconstructed in step 3 with its detailed albedo map and depth shift map to synthesize a fine-grained 3D face. Render the 3D face using a differentiable renderer to generate a fine-grained face image.

Images