A method for reconstructing 3D head model based on single-view face image

Abstract

The present invention provides a method for reconstructing a three-dimensional head model based on a single-view facial image, which relates to the field of three-dimensional head model reconstruction and specifically includes the following steps: obtaining a single-view input frontal face image of the person to be reconstructed, using a posture estimation model to estimate the camera pose parameters of the frontal face in the input image; using a three-dimensional diffusion model to estimate images of the head at other angles based on the frontal face image and the camera pose parameters corresponding to the frontal face image; using a three-dimensional generative adversarial network to decode the latent vector representing the head model, and using the frontal face image, images at other angles, and the corresponding camera parameters as supervision to iteratively optimize the latent vector; rendering a three-dimensional head image at any angle based on the optimized latent vector, and extracting the corresponding three-dimensional head geometric model. The technical solution of the present invention overcomes the problem in the prior art that only single-view images are used as input and that a high-quality three-dimensional head model that maintains spatial consistency at any angle cannot be generated.

Description

Technical Field

[0001] The present invention relates to the technical field of three-dimensional head model reconstruction, and in particular to a three-dimensional head model reconstruction method based on a single-view face image. Background Art

[0002] Research in 3D human reconstruction aims to reconstruct a virtual 3D human model from images or video data captured from different angles, simulating the appearance and details of a real human body. Reconstructing the human head is an integral part of this task and often determines the final quality of the entire reconstruction.

[0003] To generate a detailed 3D human head model, traditional modeling methods require learning parameterized geometry and textures from large amounts of 3D scan data, then representing it using voxels such as point clouds and triangulated meshes. Consequently, generating high-fidelity 3D digital portraits often requires expensive equipment and harsh shooting environments. This not only makes virtual reality (VR) inaccessible to ordinary people, but also violates the requirements for fast, low-cost, and high-precision 3D portrait reconstruction. In contrast, 3D digital human reconstruction technology based on single-view image input does not rely on expensive equipment or harsh environments, significantly reducing the cost of morphological capture and increasing the speed of human modeling. This technology has broad application in downstream fields such as biomedicine, film and television production, and VR. Image reconstruction using single-view input involves modeling the entire 3D human head model using only a single frontal face photo. However, a single frontal face input image often fails to capture features and information from both the face and the side, as well as the back of the head. This makes it difficult for traditional methods to reconstruct a complete head model, such as hairstyle and back of the head. Therefore, generating high-quality 3D head models that maintain spatial consistency from any angle using only a single-view image as input is a key research topic.

[0004] Therefore, there is a need for a method that can generate a high-quality 3D human head model that remains spatially consistent at any angle using only a single-view image as input. Summary of the Invention

[0005] The main purpose of the present invention is to provide a method for reconstructing a three-dimensional human head model based on a single-view facial image, so as to solve the problem in the prior art that only a single-view image is used as input and a high-quality three-dimensional human head model that maintains spatial consistency at any angle cannot be generated.

[0006] To achieve the above object, the present invention provides a method for reconstructing a 3D head model based on a single-view face image, which specifically comprises the following steps:

[0007] S1, obtain a single-view input frontal face image of the person to be reconstructed, and use a pose estimation model to estimate the camera pose parameters of the frontal face in the input image.

[0008] S2, based on the frontal face image and the camera pose parameters corresponding to the frontal face image, uses a three-dimensional diffusion model to estimate images of the head at other angles.

[0009] S3 uses a 3D generative adversarial network to decode the latent vector representing the human head model, and uses the frontal face image, images from other angles, and the corresponding camera parameters as supervision to iteratively optimize the latent vector.

[0010] S4, based on the optimized latent vector, renders a 3D human head image at any angle and extracts the corresponding 3D human head geometric model.

[0011] Furthermore, step S1 specifically includes the following steps:

[0012] S1.1, obtain the RGB image of the front of the person to be reconstructed, and use the 3DDFA model to transform the input image I gt Perform cropping and align the image coordinate origin with the center of the face.

[0013] S1.2, according to image I gt The orientation of the mid-face, continue to use the 3DDFA model E to estimate the homogeneous matrix c of the front face relative to the coordinate system front :

[0014] c front =E(I gt ).

[0015] S1.3, add the rotation matrix R to the homogeneous matrix to obtain the camera pose matrix at any angle of the head, and select the camera parameter c of the back of the head opposite to the front back The camera pose matrix at any angle of the head:

[0016]

[0017] c back =R×c front .

[0018] Furthermore, step S2 specifically includes the following steps:

[0019] S2.1, given k camera perspectives {π1,π2,...,π k} and given image y, the joint distribution p(z) is composed of RGB images:

[0020] p(z)=p c (x 1:k |y);

[0021] Among them, p crepresents the RGB image x observed by the 3D diffusion model conditioned on the image y 1:k The joint distribution of .

[0022] S2.2, fitting p by learning the three-dimensional diffusion model f c , thereby synthesizing the k camera perspectives {π1,π2,...,π k} under the corresponding RGB image (x 1:k ):

[0023] (x 1:k )=f(y,π 1:k );

[0024] S2.3, x 1:k Expressed as a Markov chain in the diffusion model:

[0025]

[0026] in, represents the random Gaussian noise from the k camera perspectives in the Tth time step, P(·) is the three-dimensional diffusion model distribution, t represents the time step, represents the RGB image from the k-camera perspective in the t-th time step, ∏ t (·) represents the cumulative multiplication of t time steps.

[0027] Furthermore, step S3 specifically includes the following steps:

[0028] S3.1, randomly sampling Gaussian noise from a latent space containing arbitrary Gaussian noise Will and the camera parameters c under the input viewing angle 1:k Splice and get

[0029] S3.2, by mapping the network M Mapped to an intermediate latent vector w.

[0030] S3.3, use the 3D generative adversarial network G to decode the intermediate latent vector w to obtain the generated image under the input perspective

[0031] S3.4, Calculate the real image and generate images The mean square error L mse and perceptual loss L p :

[0032]

[0033]

[0034] Where 1:k is the input image index and F(·) is the VGG16 neural network used to extract deep features of the image.

[0035] S3.5, calculate the total loss L:

[0036] L=L p +L mse .

[0037] The present invention has the following beneficial effects:

[0038] The present invention proposes a method for reconstructing a three-dimensional human head model from a single-view input based on a three-dimensional diffusion model and a three-dimensional generative adversarial network. The three-dimensional diffusion model is used to provide spatial information from other viewpoints, and a three-dimensional generative adversarial network is used to decode the latent vector. Finally, the mean square error and perceptual loss are used to iteratively optimize the latent vector, so that the latent vector can better represent the spatial features of the three-dimensional human head, thereby reconstructing a high-quality, high-fidelity three-dimensional human head model with geometric texture consistency under the premise of a single-view input image. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the specific embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0040] Figure 1 The flowchart of the method for reconstructing a three-dimensional head model based on a single-view face image of the present invention is shown. DETAILED DESCRIPTION

[0041] The technical solution of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0042] like Figure 1 The method for reconstructing a 3D head model based on a single-view face image specifically includes the following steps:

[0043] S1, obtain a single-view input frontal face image of the person to be reconstructed, and use a pose estimation model to estimate the camera pose parameters of the frontal face in the input image.

[0044] S2, based on the frontal face image and the camera pose parameters corresponding to the frontal face image, uses a three-dimensional diffusion model to estimate images of the head at other angles.

[0045] S3 uses a 3D generative adversarial network to decode the latent vector representing the human head model, and uses the frontal face image, images from other angles, and the corresponding camera parameters as supervision to iteratively optimize the latent vector.

[0046] S4, based on the optimized latent vector, renders a 3D human head image at any angle and extracts the corresponding 3D human head geometric model.

[0047] Specifically, step S1 includes the following steps:

[0048] S1.1, obtain the RGB image of the front of the person to be reconstructed, and use the 3DDFA model to transform the input image I gt Perform cropping and align the image coordinate origin with the center of the face. The 3D Dense Face Alignment Model (3DDFA) is a method for estimating the camera pose parameters of the face facing the camera based on the input image.

[0049] S1.2, according to image I gt The orientation of the mid-face, continue to use the 3DDFA model E to estimate the homogeneous matrix c of the front face relative to the coordinate system front : The matrix size is 4×6, the homogeneous matrix c front Contains external information such as the camera's spatial information and internal information such as focal length;

[0050] c front =E(I gt ).

[0051] S1.3, add the rotation matrix R to the homogeneous matrix to obtain the camera pose matrix at any angle of the head. In order to reduce the interference of redundant images on the head modeling, we select the camera parameter c of the back of the head opposite to the front after verification. back .

[0052]

[0053] c back =R×c front .

[0054] Specifically, step S2 includes the following steps:

[0055] S2.1, given k camera perspectives {π1,π2,...,π k} and given image y, the joint distribution p(z) is composed of RGB images:

[0056] p(z)=p c (x1:k |y);

[0057] Among them, p c represents the RGB image x observed by the 3D diffusion model conditioned on the image y 1:k The joint distribution of .

[0058] S2.2, fitting p by learning the three-dimensional diffusion model f c , thereby synthesizing the k camera perspectives {π1,π2,...,π k} under the corresponding RGB image (x 1:k ):

[0059] (x 1:k )=f(y,π 1:k );

[0060] S2.3, x 1:k Expressed as a Markov chain in the diffusion model:

[0061]

[0062] in, represents the random Gaussian noise from the k camera perspectives in the Tth time step, P(·) is the three-dimensional diffusion model distribution, t represents the time step, represents the RGB image from the k-camera perspective in the t-th time step, ∏ t (·) represents the cumulative multiplication of t time steps. The RGB image and normal map under the target perspective can be sampled in the Markov chain.

[0063] Specifically, step S3 includes the following steps:

[0064] S3.1, randomly sampling Gaussian noise from a latent space containing arbitrary Gaussian noise Will and the camera parameters c under the input viewing angle 1:k Splice and get

[0065] S3.2, by mapping the network M Mapped to an intermediate latent vector w.

[0066] S3.3, use the 3D generative adversarial network G to decode the intermediate latent vector w to obtain the generated image under the input perspective

[0067] S3.4, Calculate the real image and generate images The mean square error L mse and perceptual loss L p :

[0068]

[0069]

[0070] Where 1:k is the input image index, and F(·) is the VGG16 neural network used to extract deep features of the image. The VGG16 network is a deep convolutional neural network architecture constructed with continuous small convolution kernels and pooling layers, with a network depth of 16 layers.

[0071] S3.5, calculate the total loss L:

[0072] L=L p +L mse .

[0073] The iterative optimization process of the intermediate latent vector w is supervised by the total loss L, so that the generated image decoded by w Gradual Approximation

[0074] Specifically, the specific process of step 4 is:

[0075] Based on the intermediate latent variable w optimized in step 3, a voxel-level reconstruction algorithm (Marching cubes) is used to extract the geometric model of the 3D human head.

[0076] Based on the intermediate latent variable w optimized in step 3, a 3D human head image is generated from any perspective.

[0077] The present invention is implemented under the PyTorch framework and uses the Adam algorithm to optimize the model with a maximum number of iterations of 1000.

[0078] In order to verify the feasibility and superiority of the present invention, the following comparative experiments were conducted on the FFHQ dataset and the K-Hairs dataset.

[0079] The most advanced PanoHead method was selected for the task of generating a 3D human head from a single-view input. The reconstruction results were compared with the recognition results of the present invention, as shown in Table 1. The PanoHead method only uses a single-view image as input and does not use a 3D diffusion model to expand the spatial information contained in the input.

[0080] This paper uses three evaluation metrics, mean squared error (MSE), perceptual loss (Perception), and Fréchet inception distance (FID), to evaluate the trained model. MSE and Perception are used to assess the difference between the generated frontal face image and the real frontal face image; smaller values ​​represent higher model accuracy. FID is divided into two parts, FID-front and FID-back. FID-front evaluates the mean and covariance of the generated frontal face image and the input real face image in the deep feature space, and then calculates the Fréchet distance between the two feature Gaussian distributions. Secondly, because the FFHQ dataset does not contain real images of the back of the head, while the K-Hairs dataset only contains images of the back of the head, it is impossible to compare and evaluate the generated results of the back of the head with the real data at the pixel level. To address this issue, the K-Hairs test set, PanoHead, and the results generated by the present invention are divided into three groups based on the yaw angle: 90°≤yaw<180°, yaw=180°, and 180°<yaw≤270°. The FID of each group is calculated according to the corresponding angle. Finally, the FID values ​​of the three groups are averaged to obtain FID-back, which is used as the final indicator for evaluating the quality of the back of the head image. The smaller the FID value, the higher the model accuracy. Among the above evaluation indicators, the mean square error (MSE), average perceptual loss (Perception), and FID-front are performed on the FFHQ dataset, and the FID-back is performed on the K-Hairs dataset.

[0081] Table 1 Comparison of the generation results between the proposed method and PanoHead on the FFHQ and K-Hairs datasets

[0082]

[0083] As can be seen from Table 1, the mean square error (MSE), average perceptual loss (Perception) and FID-front of the proposed method on the FFHQ dataset reached 1.8, 6.1 and 13.5, which are lower than 2.1, 6.9 and 14.0 of PanoHead. The FID-back of the proposed method on the K-Hairs dataset also achieved the best result of 52.1, which is lower than 64.9 of PanoHead, effectively improving the reconstruction quality of the 3D human head model.

[0084] Of course, the above description is not a limitation of the present invention, and the present invention is not limited to the above examples. Changes, modifications, additions or substitutions made by technicians in this technical field within the essential scope of the present invention should also fall within the scope of protection of the present invention.

Claims

1. A method for reconstructing a three-dimensional head model based on a single-view face image, characterized in that: The specific steps include: S1, obtain a single-view input frontal face image of the person to be reconstructed, and use a pose estimation model to estimate the camera pose parameters of the frontal face in the input image; S2, based on the frontal face image and the camera pose parameters corresponding to the frontal face image, uses a three-dimensional diffusion model to estimate the image of the head at other angles; S3 uses a 3D generative adversarial network to decode the latent vector representing the human head model, and uses the frontal face image, images from other angles, and corresponding camera parameters as supervision to iteratively optimize the latent vector; S4, based on the optimized latent vector, renders a 3D human head image at any angle and extracts the corresponding 3D human head geometric model; Step S3 specifically includes the following steps: S3.1, randomly sampling Gaussian noise from a latent space containing arbitrary Gaussian noise Will and the camera parameters c under the input viewing angle 1:k Splice and get S3.2, by mapping the network M Mapped to an intermediate latent vector w; S3.3, use the 3D generative adversarial network G to decode the intermediate latent vector w to obtain the generated image under the input perspective S3.4, Calculate the real image and generate images The mean square error L mse and perceptual loss L p : Where 1:k is the input image index, F(·) is the VGG16 neural network used to extract deep features of the image; S3.5, calculate the total loss L: L=L p +L mse 。 2. The method for reconstructing a 3D head model based on a single-view face image according to claim 1, characterized in that: Step S1 specifically includes the following steps: S1.1, obtain the RGB image of the front of the person to be reconstructed, and use the 3DDFA model to transform the input image I gt Perform cropping and align the image coordinate origin with the center of the face; S1.2, according to image I gt The orientation of the mid-face, continue to use the 3DDFA model E to estimate the homogeneous matrix c of the front face relative to the coordinate system front : c front =E(I gt ); S1.3, add the rotation matrix R to the homogeneous matrix to obtain the camera pose matrix at any angle of the head, and select the camera parameter c of the back of the head opposite to the front back The camera pose matrix at any angle of the head: c back =R×c front 。 3. The method for reconstructing a 3D head model based on a single-view facial image according to claim 1, wherein: Step S2 specifically includes the following steps: S2.1, given k camera perspectives {π1,π2,...,π k } and given image y, the joint distribution p(z) is composed of RGB images: p(z)=p c (x 1:k |y); Among them, p c represents the RGB image x observed by the 3D diffusion model conditioned on the image y 1:k The joint distribution of S2.2, fitting p by learning the three-dimensional diffusion model f c , thereby synthesizing the k camera perspectives {π1,π2,...,π k } under the corresponding RGB image (x 1:k ): (x 1:k )=f(y,π 1:k ); S2.3, x 1:j Expressed as a Markov chain in the diffusion model: in, represents the random Gaussian noise from the k camera perspectives in the Tth time step, P(·) is the three-dimensional diffusion model distribution, t represents the time step, represents the RGB image from the k-camera perspective in the t-th time step, ∏ t (·) represents the cumulative multiplication of t time steps.

Images