Multi-view three-dimensional head model reconstruction method based on back-side head completion

Abstract

The invention provides a multi-view three-dimensional head model reconstruction method based on back side completion, and relates to the field of three-dimensional head model reconstruction, and the method specifically comprises the following steps: obtaining a multi-view data set of a target person, and employing a posture estimation model to estimate the camera posture parameters of the front face orientation of an image in the multi-view data set; iteratively optimizing the three-dimensional generative adversarial network by taking the multi-view images and the corresponding camera pose parameters as supervision information; rendering an image of the back side part based on the optimized three-dimensional generative adversarial network, and taking the image and the corresponding camera parameters as a back side view angle data set; and training a high-quality three-dimensional head model by taking the back-head view data set and the original multi-view data set together as supervision information. According to the technical scheme, the problems that in the prior art, back-side view angle data are ignored, so that the integrity and detail reduction of a reconstruction model are obviously insufficient, and particularly the geometrical shape and texture details of a back-side area are poor are solved.

Description

Technical Field

[0001] The present invention relates to the field of three-dimensional head model reconstruction, and in particular to a multi-view three-dimensional head model reconstruction method based on back of head completion. Background Art

[0002] The reconstruction of a 3D head model is one of the core tasks of digital human 3D reconstruction research. Its accuracy and detail restoration capabilities directly affect the realism and naturalness of digital humans. Traditional 3D head model reconstruction methods mainly rely on multi-view images, depth sensors, or laser scanners to obtain 3D data. Although these methods can generate high-precision 3D models, they have limitations such as high equipment cost, complex operation, and strict environmental requirements, making it difficult to meet the convenience and universality requirements in practical applications.

[0003] In recent years, 3D reconstruction methods based on a single or a small number of images have gradually become a research hotspot, especially the use of deep learning technology for model reconstruction. These methods can infer 3D geometric information from limited input images by learning the shape and texture features of the human head in large-scale data sets. However, existing methods mainly rely on frontal or side image information, ignoring the difficult-to-obtain perspective data such as the back of the head, resulting in obvious deficiencies in the integrity and detail restoration of the reconstructed model, especially in the geometric shape and texture details of the back of the head area.

[0004] In practical applications, the integrity of the back of the head area is crucial for the reconstruction of a three-dimensional head model. For example, in virtual reality or film and television production, users or viewers may observe the model from any angle. If the back of the head area is missing or distorted, it will seriously affect the visual effect and user experience. In addition, the details of the hair, ears, etc. in the back of the head area are complex and diverse, which further increases the difficulty of reconstruction. The back of the head completion technology can effectively fill in the missing parts in multi-view data by generating or inferring the geometric and texture information of the back of the head area. Therefore, how to use the back of the head completion technology to improve the reconstruction effect of the three-dimensional head model has become a current research hotspot.

[0005] Therefore, a method for reconstructing a high-precision three-dimensional human head model based on back of the head completion is needed. Summary of the invention

[0006] The main purpose of the present invention is to provide a multi-view three-dimensional human head model reconstruction method based on back of the head completion, so as to solve the problem that the prior art ignores the difficult-to-obtain view data such as the back of the head, resulting in obvious deficiencies in the integrity and detail restoration of the reconstructed model, especially the poor performance in the geometric shape and texture details of the back of the head area.

[0007] To achieve the above object, the present invention provides a multi-view 3D head model reconstruction method based on back of head completion, which specifically comprises the following steps: S1, obtain a multi-view dataset of the target person, and use a posture estimation model to estimate the camera pose parameters of the front face orientation of the image in the multi-view dataset.

[0008] S2, taking multi-view images and corresponding camera pose parameters as supervision information, iteratively optimizes the 3D generative adversarial network.

[0009] S3, based on the optimized 3D generative adversarial network, renders an image of the back of the head, and uses the image and the corresponding camera parameters as the back of the head perspective dataset.

[0010] S4, uses the back of the head perspective dataset and the original multi-view dataset as supervision information to train a high-quality 3D human head model.

[0011] Furthermore, step S1 specifically includes the following steps: S1.1, obtain a multi-view dataset of the target person, perform face detection on the image, and select and retain the valid face area Images, cropped using the face alignment model 3DDFA Multi-view images , so that the multi-view image The coordinate origin is aligned with the center of the face.

[0012] S1.2, based on multi-view images The face orientation information in the image is estimated using the face alignment model 3DDFA. Camera parameters corresponding to each face : ; in, It is the face alignment model 3DDFA.

[0013] Furthermore, step S2 specifically includes the following steps: S2.1, randomly sampling Gaussian noise from a latent space containing arbitrary Gaussian noise , and the Gaussian noise Camera parameters under input viewing angle Combined, by mapping the network Map randomly sampled Gaussian noise to an intermediate latent vector , using a 3D generative adversarial network Decoding the intermediate latent vector , get the generated image under the input perspective : .

[0014] S2.2, Calculation of multi-view images And generate images The mean square error loss and perceived loss : ; ; in, is the input image index, VGG16 neural network is used to extract deep features of images.

[0015] S2.3, based on mean square error loss and perceived loss , and the regularization term jointly optimizes the intermediate latent vector , updating the intermediate latent vector in each iteration , so that the generated image With multi-view images Gradually approaching.

[0016] S2.4, in the intermediate latent vector After the optimization is completed, the intermediate latent vector is fixed And optimize the 3D generative adversarial network Parameters.

[0017] Furthermore, step S3 specifically includes the following steps: S3.1, determine the camera internal parameters And render the k camera positions and orientations of the back of the head, and according to the camera position and camera orientation Calculate the corresponding camera extrinsic matrix : ; in, , is the focal length, are the principal point coordinates of the image; ; in, , is the camera’s direction; , is the right direction of the camera; , is the upward direction of the camera; is the norm, , .

[0018] S3.2, the optimized intermediate latent vector k camera parameters corresponding to the back of the head perspective Combined with fine-tuned 3D generative adversarial network Decode and get the generated image from the back of the head : .

[0019] S3.3, will generate an image , generate the camera extrinsic parameters and camera intrinsic parameters corresponding to the image, and combine them together in the format of the original multi-view dataset to obtain the back of the head perspective dataset.

[0020] Furthermore, step S4 specifically includes the following steps: S4.1, bind the initialized Gaussian point cloud and the parameterized head model FLAME as the Gaussian character 3D head model, and use Gaussian rendering combined with the camera parameters of the input perspective to obtain the rendered image of the corresponding perspective .

[0021] S4.2, construct coordinate transformation matrix, including rotation matrix , scaling matrix and translation matrix ; When the input is the back of the head dataset, all mesh vertices of the Gauss character 3D head model are The coordinates are transformed to the same position as the back of the head dataset before rendering: ; in, , , , , is the number of vertices, For the rendered .

[0022] S4.3, the back of the head view dataset and the multi-view dataset are used as real images , calculate the structural similarity loss between the real image and the rendered image : ; in, is the structural similarity index, is the weight coefficient of the structural similarity loss.

[0023] S4.4, in each round of iteration, the structural similarity loss between the real image and the rendered image and the geometric constraints between the Gaussian points are used to adjust the parameters of the parameterized head model and the Gaussian point cloud. The back of the head perspective dataset additionally optimizes the coordinate transformation matrix so that the rendered image With real image Gradually approaching.

[0024] The present invention has the following beneficial effects: The present invention proposes a solution to the problem of missing or distorted data in the back of the head area in the traditional multi-view reconstruction method. Then, by constructing a coordinate transformation matrix, the three-dimensional space offset problem between the back of the head perspective data set and the parameterized head model is solved, ensuring the consistency of the rendering perspective and the real perspective, thereby reconstructing a high-quality, high-fidelity three-dimensional head model with geometric texture consistency under the premise of multi-view input images. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the specific implementation of the present invention or the technical solutions in the prior art, the following will briefly introduce the drawings required for the specific implementation or the prior art description. Obviously, the drawings described below are some implementations of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings: Figure 1 The flowchart of the method for reconstructing a multi-view three-dimensional head model based on back of the head completion of the present invention is shown.

[0026] Figure 2 An image of the back of the head of a Gaussian character 3D head model trained using a multi-view dataset is shown.

[0027] Figure 3 An image of the back of the head of a Gaussian character three-dimensional head model trained using the method provided by the present invention is shown. DETAILED DESCRIPTION

[0028] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0029] like Figure 1 A multi-view 3D head model reconstruction method based on back head completion is shown, which specifically includes the following steps: S1, obtain a multi-view dataset of the target person, and use a posture estimation model to estimate the camera pose parameters of the front face orientation of the image in the multi-view dataset.

[0030] S2, taking multi-view images and corresponding camera pose parameters as supervision information, iteratively optimizes the 3D generative adversarial network.

[0031] S3, based on the optimized 3D generative adversarial network, renders an image of the back of the head, and uses the image and the corresponding camera parameters as the back of the head perspective dataset.

[0032] S4, uses the back of the head perspective dataset and the original multi-view dataset as supervision information to train a high-quality 3D human head model.

[0033] Specifically, step S1 includes the following steps: S1.1, obtain a multi-view dataset of the target person, perform face detection on the image, and select and retain the valid face area Images, cropped using the face alignment model 3DDFA Multi-view images , so that the multi-view image The coordinate origin is aligned with the center of the face.

[0034] S1.2, based on multi-view images The orientation information of the face in the image is estimated using the face alignment model 3DDFA The camera parameters corresponding to the face (including camera intrinsics and camera extrinsics) : ; in, It is the face alignment model 3DDFA.

[0035] Specifically, step S2 includes the following steps: S2.1, randomly sampling Gaussian noise from a latent space containing arbitrary Gaussian noise , and the Gaussian noise Camera parameters under input viewing angle (including camera internal parameters and camera external parameters) combined through the mapping network Map randomly sampled Gaussian noise to an intermediate latent vector , using a 3D generative adversarial network Decoding the intermediate latent vector , get the generated image under the input perspective : .

[0036] S2.2, Calculation of multi-view images And generate images The mean square error loss and perceived loss : ; ; in, is the input image index, VGG16 neural network is used to extract deep features of images.

[0037] S2.3, based on mean square error loss and perceived loss , and the regularization term jointly optimizes the intermediate latent vector , updating the intermediate latent vector in each iteration , so that the generated image With multi-view images Gradually approaching.

[0038] S2.4, in the intermediate latent vector After the optimization is completed, the intermediate latent vector is fixed And optimize the 3D generative adversarial network Parameters.

[0039] Specifically, step S3 includes the following steps:

[0040] S3.1, determine the camera internal parameters And render the k camera positions and orientations of the back of the head, and according to the camera position and camera orientation Calculate the corresponding camera extrinsic matrix : ; in, , is the focal length, are the principal point coordinates of the image; ; in, , is the camera’s direction; , is the right direction of the camera; , is the upward direction of the camera; is the norm, , .

[0041] S3.2, the optimized intermediate latent vector k camera parameters corresponding to the back of the head perspective (including camera intrinsics and camera extrinsics) combined with a fine-tuned 3D generative adversarial network Decode and get the generated image from the back of the head : .

[0042] S3.3, will generate an image , generate the camera extrinsic parameters and camera intrinsic parameters corresponding to the image, and combine them together in the format of the original multi-view dataset to obtain the back of the head perspective dataset.

[0043] Specifically, step S4 includes the following steps: S4.1, bind the initialized Gaussian point cloud and the parameterized head model FLAME as the Gaussian character 3D head model, and use Gaussian rendering combined with the camera parameters of the input perspective to obtain the rendered image of the corresponding perspective .

[0044] S4.2, to solve the problem of the three-dimensional space offset between the back head perspective dataset generated by the three-dimensional generative adversarial network and the parameterized head model, construct the coordinate transformation matrix, including the rotation matrix , scaling matrix and translation matrix ; When the input is the back of the head dataset, all mesh vertices of the Gauss character 3D head model are The coordinates are transformed to the same position as the back of the head dataset before rendering: ; in, , , , , is the number of vertices, For the rendered .

[0045] S4.3, the back of the head view dataset and the multi-view dataset are used as real images , calculate the structural similarity loss between the real image and the rendered image : ; in, is the structural similarity index, is the weight coefficient of the structural similarity loss.

[0046] S4.4, in each round of iteration, the structural similarity loss between the real image and the rendered image and the geometric constraints between the Gaussian points are used to adjust the parameters of the parameterized head model and the Gaussian point cloud. The back of the head perspective dataset additionally optimizes the coordinate transformation matrix so that the rendered image With real image Gradually approaching.

[0047] The present invention is implemented under the PyTorch framework, and the Adam algorithm is used to optimize the model. The maximum number of iterations of the three-dimensional generative adversarial network is 1000, and the maximum number of iterations of the Gaussian character three-dimensional head model is 600000.

[0048] After introducing the back of the head completion dataset, the reconstruction completeness of the Gaussian character 3D head model without real back of the head data is significantly improved, and details such as hair and ears in the back of the head area that do not appear in the dataset can be generated. The back of the head part of the Gaussian character 3D head model trained only with the multi-view dataset is compared with the back of the head part of the Gaussian character 3D head model trained jointly with the back of the head view dataset and the multi-view dataset. Figure 2 and Figure 3 shown.

[0049] Of course, the above description is not a limitation of the present invention, and the present invention is not limited to the above examples. Changes, modifications, additions or substitutions made by technicians in this technical field within the essential scope of the present invention should also fall within the protection scope of the present invention.

Claims

1. A multi-view 3D head model reconstruction method based on back of head completion, characterized in that: The specific steps include: S1, obtain a multi-view dataset of the target person, and use a pose estimation model to estimate the camera pose parameters of the front face orientation of the image in the multi-view dataset; S2, taking multi-view images and corresponding camera pose parameters as supervision information, iteratively optimizes the 3D generative adversarial network; S3, based on the optimized 3D generative adversarial network, renders the image of the back of the head, and uses the image and the corresponding camera parameters as the back of the head perspective dataset; S4, uses the back of the head perspective dataset and the original multi-view dataset as supervision information to train a high-quality 3D human head model.

2. The method for reconstructing a multi-view 3D human head model based on back of head completion according to claim 1, characterized in that: Step S1 specifically includes the following steps: S1.1, obtain a multi-view dataset of the target person, perform face detection on the image, and select and retain the valid face area Images, cropped using the face alignment model 3DDFA Multi-view images , so that the multi-view image The coordinate origin is aligned with the center of the face; S1.2, based on multi-view images The face orientation information in the image is estimated using the face alignment model 3DDFA. Camera parameters corresponding to each face : ； in, It is the face alignment model 3DDFA.

3. The method for reconstructing a multi-view 3D head model based on back of head completion according to claim 1, characterized in that: Step S2 specifically includes the following steps: S2.1, randomly sample Gaussian noise from a latent space containing arbitrary Gaussian noise , and the Gaussian noise Camera parameters under input viewing angle Combined, by mapping the network Map randomly sampled Gaussian noise to an intermediate latent vector , using a 3D generative adversarial network Decoding the intermediate latent vector , and get the generated image under the input perspective : ； S2.2, Calculation of multi-view images And generate images The mean square error loss and perceived loss : ； ； in, is the input image index, VGG16 neural network for extracting deep features of images; S2.3, based on mean square error loss and perceived loss , and the regularization term jointly optimizes the intermediate latent vector , updating the intermediate latent vector in each iteration , so that the generated image With multi-view images gradually approaching; S2.4, in the intermediate latent vector After the optimization is completed, the intermediate latent vector is fixed And optimize the 3D generative adversarial network Parameters.

4. The method for reconstructing a multi-view 3D head model based on back of head completion according to claim 1, characterized in that: Step S3 specifically includes the following steps: S3.1, determine the camera internal parameters And render the k camera positions and orientations of the back of the head, and according to the camera position and camera orientation Calculate the corresponding camera extrinsic matrix : ； in, , is the focal length, are the principal point coordinates of the image; ； in, , is the camera’s direction; , is the right direction of the camera; , is the upward direction of the camera; is the norm, , ; S3.2, the optimized intermediate latent vector k camera parameters corresponding to the back of the head perspective Combined with fine-tuned 3D generative adversarial network Decode and get the generated image from the back of the head : ； S3.3, will generate an image , generate the camera extrinsic parameters and camera intrinsic parameters corresponding to the image, and combine them together in the format of the original multi-view dataset to obtain the back of the head perspective dataset.

5. The method for reconstructing a multi-view 3D head model based on back of head completion according to claim 1, characterized in that: Step S4 specifically includes the following steps: S4.1, bind the initialized Gaussian point cloud and the parameterized head model FLAME as the Gaussian character 3D head model, and use Gaussian rendering combined with the camera parameters of the input perspective to obtain the rendered image of the corresponding perspective ; S4.2, construct coordinate transformation matrix, including rotation matrix , scaling matrix and translation matrix ; When the input is the back of the head dataset, all mesh vertices of the Gauss character 3D head model are The coordinates are transformed to the same position as the back of the head dataset before rendering: ； in, , , , , is the number of vertices, For the rendered ; S4.3, the back of the head view dataset and the multi-view dataset are used as real images , calculate the structural similarity loss between the real image and the rendered image : ； in, is the structural similarity index, is the weight coefficient of structural similarity loss; S4.4, in each round of iteration, the structural similarity loss between the real image and the rendered image and the geometric constraints between the Gaussian points are used to adjust the parameters of the parameterized head model and the Gaussian point cloud. The back of the head perspective dataset additionally optimizes the coordinate transformation matrix so that the rendered image With real image Gradually approaching.

Images