Densification method and device for single-video 3DGS human head reconstruction and storage medium

Abstract

The invention discloses a densification method and device for single-video 3DGS head reconstruction and a storage medium, and the method comprises the steps: carrying out the calculation based on a video sample, and obtaining a sparse point cloud covering three-dimensional position information; initializing the sparse point cloud to obtain a 3D Gaussian ball; according to the method, 3D GS model training is carried out based on a 3D Gaussian ball, in the adaptive densification control process of 3D GS model training, densification indexes are introduced into a sampling probability function, the best densification path is automatically updated and searched in the 3D GS model training process, and a 3D GS model is obtained after training is completed. In the adaptive densification control process of 3DGS model training, the densification index is introduced into the sampling probability function, and the addition and deletion process of the 3D Gaussian sphere is upgraded to a continuous differentiable full-automatic process, so that the change of data in the training process can be sensed more sensitively; a non-differentiating processing mode in which a threshold value needs to be set manually is replaced, and manual interference is avoided.

Description

Technical Field

[0001] The present invention relates to the technical field of digital human, and specifically provides a densification method, device and storage medium for single-video 3DGS human head reconstruction. Background Art

[0002] The emergence of 3D Gaussian Splatting (3DGS) has greatly accelerated the rendering speed of new view synthesis. Unlike neural implicit representations such as Neural Radiance Field (NeRF) that represent 3D scenes with position and viewpoint-conditioned neural networks, 3DGS uses a set of Gaussian ellipsoids to model the scene, thereby achieving efficient rendering by rasterizing the Gaussian ellipsoids into images.

[0003] For head modeling using 3DGS, MonoGaussianAvatar first applies 3DGS for dynamic head reconstruction, using canonical space modeling and deformation prediction. In addition, PSAvatar introduces an explicit Flame facial model to initialize the Gaussian, which can capture high-fidelity facial geometry and even complex volumetric objects (such as glasses). GaussianHead uses a three-plane representation and motion field to simulate a geometrically changing head in continuous motion and render rich textures, including skin and hair. In order to more easily control head expressions, GaussianAvatars introduces geometric priors (Flame parameterized facial model) into 3DGS, binds the Gaussian to an explicit mesh, and optimizes the parameters of the Gaussian ellipsoid.

[0004] Rig3DGS uses learnable deformations to provide stability and generalization to novel expressions, head poses, and viewing directions, enabling controllable portraits on portable devices. HeadGas endows 3DGS with a set of latent feature-based foundations weighted by the expression vectors of 3DMMs, enabling real-time animated head reconstruction.

[0005] FlashAvatar further embeds the uniform 3DGS field into the parameterized facial model and learns additional spatial offsets to capture facial details, successfully pushing the rendering speed to 300FPS. In order to synthesize high-resolution results, GaussianHead Avatar uses a super-resolution network to achieve high-fidelity avatar learning. GaussianHair combines the Marschner hair model with UE4's real-time hair rendering for the first time to create a Gaussian hair scattering model. It captures complex hair geometry and appearance for fast rasterization and volume rendering, enabling applications including editing and relighting.

[0006] However, regardless of the technique used for head modeling using 3DGS, 3DGS training relies heavily on the densification process. The original adaptive densification strategy was to accumulate position gradients and manually set thresholds. When the accumulated gradient of a Gaussian sphere is greater than a certain threshold, the Gaussian sphere will be cloned or split.

[0007] Some studies have adopted the design of the original 3DGS, while others have chosen to subdivide the grid and then assign Gaussian spheres to new vertices. Other studies have simply randomly assigned new Gaussian spheres near the current Gaussian spheres. However, all of these strategies will generate redundant Gaussian spheres, which seriously affects rendering efficiency. Moreover, the setting of the threshold as a hyperparameter affects the automation process of 3DGS. Different characters or task scenarios are likely to have different optimal thresholds, and it is more likely that a fixed threshold is inappropriate.

[0008] In view of this, the present invention patent is proposed. Summary of the invention

[0009] In view of the above technical problems, the present invention proposes a densification method, device and storage medium for single-video 3DGS head reconstruction, which transforms the 3DGS densification process from a process requiring manual intervention to a fully automated and differentiable process.

[0010] Specifically, the following technical solutions are adopted:

[0011] In a first aspect, the present invention provides a densification method for single video 3DGS head reconstruction, comprising:

[0012] Based on the video samples, a sparse point cloud containing three-dimensional position information is calculated;

[0013] Initialize the sparse point cloud to get a 3D Gaussian sphere;

[0014] The 3DGS model is trained based on the 3D Gaussian sphere. In the adaptive densification control process of the 3DGS model training, the densification index is introduced into the sampling probability function. The best densification path is automatically updated during the 3DGS model training process, and the 3DGS model is obtained after the training is completed.

[0015] As an optional embodiment of the present invention, in a densification method for single video 3DGS head reconstruction of the present invention, the sampling probability function is:

[0016]

[0017] in, is the importance index of a certain 3D Gaussian sphere, and the denominator represents the sum of the importance indexes of all 3D Gaussian spheres of the triangle face where the certain 3D Gaussian sphere is located.

[0018] As an optional embodiment of the present invention, in a densification method for single-video 3DGS head reconstruction of the present invention, the importance indicators of the sampling probability function include: a maximum value indicator max(s), a maximum value ratio index max(s) / min(s), a parameter gradient index, and a transparency index; wherein s represents position information, that is, the gradient relative to the position coordinates (x, y, z) of the 3D Gaussian sphere.

[0019] As an optional implementation manner of the present invention, in a densification method for single-video 3DGS head reconstruction of the present invention, the sampling probability function adds different weights to different designated important areas of head reconstruction.

[0020] As an optional embodiment of the present invention, in a densification method for single video 3DGS head reconstruction of the present invention, the 3DGS model training based on the 3D Gaussian sphere includes:

[0021] For every N iterations of the 3D Gaussian sphere, the order of the spherical harmonic coefficients is increased, where N is the set value;

[0022] Randomly select a camera perspective;

[0023] Render the image, obtaining the view space point, visibility filter, and radius information;

[0024] Calculate the loss L = (1-λ)L1 + λL D-SSIM (L1 loss and L D-SSIM The weighted sum of the losses, L1 loss helps ensure pixel-level accuracy, and L D-SSIM The loss helps to maintain the overall structure and visual quality of the image), λ is the preset weight value, and back propagation is performed;

[0025] Subsequent operations via gradient-free contextual learning include:

[0026] Perform point cloud density operations based on the number of iterations:

[0027] Update the maximum radius information;

[0028] Increase and trim point cloud density based on conditions;

[0029] Update the optimizer parameters;

[0030] During the point cloud density operation, the densification index is introduced into the sampling probability function, which is:

[0031]

[0032] in, is the importance index of a certain 3D Gaussian sphere, and the denominator represents the sum of the importance indexes of all 3D Gaussian spheres of the triangle face where the certain 3D Gaussian sphere is located.

[0033] As an optional embodiment of the present invention, a densification method for single video 3DGS head reconstruction of the present invention comprises:

[0034] Shoot an image or video of the target person, input the image or video into the trained 3DGS model, and generate a 3D reconstruction model of the target person.

[0035] In a second aspect, the present invention provides a densification device for single video 3DGS head reconstruction, including a 3DGS model training module, wherein the model training module includes:

[0036] The point cloud acquisition unit calculates and obtains a sparse point cloud covering three-dimensional position information based on the video samples;

[0037] 3D Gaussian sphere generation unit, which initializes the sparse point cloud to obtain a 3D Gaussian sphere;

[0038] The 3DGS model training unit performs 3DGS model training based on the 3D Gaussian sphere. In the adaptive densification control process of the 3DGS model training, the densification index is introduced into the sampling probability function. The best densification path is automatically updated during the 3DGS model training process, and the 3DGS model is obtained after the training is completed.

[0039] As an optional embodiment of the present invention, a densification device for single-video 3DGS head reconstruction of the present invention includes a three-dimensional reconstruction module: acquiring a captured image video of a target person, inputting the image video into a 3DGS model, and generating a three-dimensional reconstruction model of the target person.

[0040] In a third aspect, the present invention provides an electronic device comprising a processor and a memory, wherein the memory is used to store a computer executable program, and when the computer program is executed by the processor, the processor executes the densification method for single-video 3DGS head reconstruction.

[0041] In a fourth aspect, the present invention provides a computer-readable recording medium storing a computer-executable program, which, when executed, implements the densification method for single-video 3DGS head reconstruction.

[0042] Compared with the prior art, the present invention has the following beneficial effects:

[0043] A densification method for single-video 3DGS head reconstruction of the present invention introduces a densification index into a sampling probability function during the adaptive densification control process of 3DGS model training, and upgrades the addition and deletion process of 3D Gaussian spheres to a continuous and differentiable fully automatic process. It can more keenly perceive the changes in data during the training process, replace the processing method that requires manual setting of thresholds and is non-differentiable, and avoid human interference. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 A flowchart of a densification method for single video 3DGS head reconstruction according to embodiment 1 of the present invention;

[0045] Figure 2 An experimental table of a densification method for single video 3DGS head reconstruction according to the first embodiment of the present invention;

[0046] Figure 3 A densified visualization effect diagram of a densification method for single video 3DGS head reconstruction according to the first embodiment of the present invention;

[0047] Figure 4 A rendering effect diagram of a densification method for single video 3DGS head reconstruction according to the first embodiment of the present invention;

[0048] Figure 5 A flowchart of a Gaussian sphere updating method for improving single-video 3DGS head reconstruction according to a second embodiment of the present invention;

[0049] Figure 6 An experimental comparison table of a Gaussian sphere update method for improving single-video 3DGS head reconstruction according to the second embodiment of the present invention;

[0050] Figure 7 A diagram showing the experimental effect of a Gaussian sphere updating method for improving single-video 3DGS head reconstruction according to the second embodiment of the present invention;

[0051] Figure 8 A flowchart of a method for reconstructing a 360-degree human head through 3DGS based on a monocular video according to a third embodiment of the present invention;

[0052] Fig. 9 An example of a training strategy for a method of reconstructing a 360-degree human head through 3DGS based on a monocular video in Embodiment 3 of the present invention;

[0053] Fig.10 Embodiment 3 of the present invention is a method for reconstructing a 360-degree human head through 3DGS based on a monocular video to reconstruct a 360-degree human head. Figure 1 ;

[0054] Fig.11Embodiment 3 of the present invention is a method for reconstructing a 360-degree human head through 3DGS based on a monocular video to reconstruct a 360-degree human head. Figure 2 ;

[0055] Fig.12 Embodiment 3 of the present invention is a method for reconstructing a 360-degree human head through 3DGS based on a monocular video to reconstruct a 360-degree human head. Figure 3 ;

[0056] Fig.13 Embodiment 3 of the present invention is a method for reconstructing a 360-degree human head through 3DGS based on a monocular video to reconstruct a 360-degree human head. Figure 4 ;

[0057] Fig.14 A schematic structural diagram of an electronic device according to a fourth embodiment of the present invention;

[0058] Fig.15 A schematic diagram of a computer-readable recording medium according to a fourth embodiment of the present invention. DETAILED DESCRIPTION

[0059] To make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be described clearly and completely in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them.

[0060] Therefore, the following detailed description of the embodiments of the present invention is not intended to limit the scope of the invention claimed for protection, but merely represents some embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0061] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features and technical solutions in the embodiments may be combined with each other.

[0062] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0063] In the description of the present invention, it should be noted that the orientation or positional relationship indicated by the terms "upper", "lower", etc. is based on the orientation or positional relationship shown in the drawings, or the orientation or positional relationship in which the invention product is usually placed when in use, or the orientation or positional relationship commonly understood by those skilled in the art. Such terms are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", etc. are only used to distinguish the description, and cannot be understood as indicating or implying relative importance.

[0064] Embodiment 1

[0065] See also Figure 1 As shown, this embodiment provides a densification method for single video 3DGS head reconstruction, including:

[0066] Based on the video samples, a sparse point cloud containing three-dimensional position information is calculated;

[0067] Initialize the sparse point cloud to get a 3D Gaussian sphere;

[0068] The 3DGS model is trained based on the 3D Gaussian sphere. In the adaptive densification control process of the 3DGS model training, the densification index is introduced into the sampling probability function. The best densification path is automatically updated during the 3DGS model training process, and the 3DGS model is obtained after the training is completed.

[0069] A densification method for single-video 3DGS head reconstruction in this embodiment introduces a densification index into the sampling probability function during the adaptive densification control process of 3DGS model training, and upgrades the process of adding and deleting 3D Gaussian balls to a continuous and differentiable fully automatic process. This can more keenly perceive the changes in data during the training process, replace the processing method that requires manual setting of thresholds and is non-differentiable, and avoid human interference.

[0070] Furthermore, in a densification method for single video 3DGS head reconstruction in this embodiment, the sampling probability function is:

[0071]

[0072] in, is the importance index of a certain 3D Gaussian sphere, and the denominator represents the sum of the importance indexes of all 3D Gaussian spheres of the triangle face where the certain 3D Gaussian sphere is located.

[0073] It can be seen from the above sampling probability function formula that the larger the importance index value of a 3D Gaussian ball is, the more important it is, the greater the sampling probability is, and vice versa. This rule is consistent with the expectations of the densification process.

[0074] So what kind of importance index works best? As an optional implementation of this embodiment, in a densification method for single-video 3DGS head reconstruction in this embodiment, the importance index of the sampling probability function includes: maximum value index max(s), maximum value ratio index max(s) / min(s), parameter gradient index, transparency index; where s represents position information, that is, the gradient relative to the position coordinates (x, y, z) of the 3D Gaussian sphere.

[0075] Specifically, using the sampling probability functions of the above four different indicators, the 3D head reconstruction results are as follows: Figure 2 As shown in the table, overall, the gradient-based head reconstruction has the best effect, while the transparency-based reconstruction has the worst effect.

[0076] In addition, the effect comparison on the test set is Figure 3 The densification visualization shown shows the distribution of splash positions during the densification process under different environments, focusing on the reconstruction effects of facial features and hair. It can be seen that the best reconstruction effect is the head reconstruction based on the gradient indicator, and the head reconstruction effect based on the maximum value indicator is the worst.

[0077] like Figure 4 As shown in the figure, the difference in effect between the "gradient-based densification strategy" and the "gradient-based densification strategy" can be observed from four angles: clarity, edge differentiation, whether hair is differentiated, and the distribution density of 3D Gaussian balls. In high-frequency areas, such as hair, eyes, nose, mouth, and eyebrows, after adding gradient-based probability sampling densification on the left, the Gaussian balls in these areas are obviously denser, which is in line with our expectations. More Gaussian balls are needed in key areas, while a small number of Gaussian balls can be used to represent smooth areas.

[0078] From the above comparison results, we can see that the 3DGS head reconstruction effect is better using the automatic continuously differentiable densification scheme.

[0079] It should be noted that this embodiment exemplifies the above four different importance indicators. Of course, the importance indicators in the sampling probability function include but are not limited to the above four indicators.

[0080] As an optional implementation of this embodiment, in a densification method for single video 3DGS head reconstruction of this embodiment, the sampling probability function adds different weights to different designated important areas of head reconstruction. In this way, the weight of the sampling probability function can be set based on each key area in the head reconstruction, and more attention is paid to the key areas in the 3D Gaussian sphere densification project of 3DGS, so that the densification effect is better and meets the needs of head reconstruction.

[0081] Specifically, in a single video 3DGS head reconstruction densification method of this embodiment, the 3DGS model training based on the 3D Gaussian sphere includes:

[0082] For every N iterations of the 3D Gaussian sphere, the order of the spherical harmonic coefficients is increased, where N is the set value;

[0083] Randomly select a camera perspective;

[0084] Render the image, obtaining the view space point, visibility filter, and radius information;

[0085] Calculate the loss L = (1-λ)L1 + λL D-SSIM (L1 loss and L D-SSIM The weighted sum of the losses, L1 loss helps ensure pixel-level accuracy, and L D-SSIM The loss helps to maintain the overall structure and visual quality of the image), λ is the preset weight value, and back propagation is performed;

[0086] Subsequent operations via gradient-free contextual learning include:

[0087] Perform point cloud density operations based on the number of iterations:

[0088] Update the maximum radius information;

[0089] Increase and trim point cloud density based on conditions;

[0090] Update the optimizer parameters;

[0091] During the point cloud density operation, the densification index is introduced into the sampling probability function, which is:

[0092]

[0093] in, is the importance index of a certain 3D Gaussian sphere, and the denominator represents the sum of the importance indexes of all 3D Gaussian spheres of the triangle face where the certain 3D Gaussian sphere is located.

[0094] A densification method for single video 3DGS head reconstruction in this embodiment includes:

[0095] Shoot an image or video of the target person, input the image or video into the 3DGS model, and generate a 3D reconstruction model of the target person.

[0096] This embodiment also provides a densification device for single video 3DGS head reconstruction, including a 3DGS model training module, wherein the model training module includes:

[0097] The point cloud acquisition unit calculates and obtains a sparse point cloud covering three-dimensional position information based on the video samples;

[0098] 3D Gaussian sphere generation unit, which initializes the sparse point cloud to obtain a 3D Gaussian sphere;

[0099] The 3DGS model training unit performs 3DGS model training based on the 3D Gaussian sphere. In the adaptive densification control process of the 3DGS model training, the densification index is introduced into the sampling probability function. The best densification path is automatically updated during the 3DGS model training process, and the 3DGS model is obtained after the training is completed.

[0100] The present embodiment provides a densification device for single-video 3DGS head reconstruction. In the adaptive densification control process of 3DGS model training, the 3DGS model training unit introduces the densification index into the sampling probability function, thereby upgrading the addition and deletion process of the 3D Gaussian sphere to a continuous and differentiable fully automatic process. This process can more keenly perceive the changes in data during the training process, replace the processing method that requires manual setting of thresholds and is non-differentiable, and avoid human interference.

[0101] Furthermore, the densification device for single-video 3DGS head reconstruction described in this embodiment includes a three-dimensional reconstruction module: acquiring a captured image video of a target person, inputting the image video into a trained 3DGS model, and generating a three-dimensional reconstruction model of the target person.

[0102] Embodiment 2

[0103] See also Figure 5 As shown, this embodiment provides a method for improving single video 3DGS head reconstruction, including a Gaussian sphere update method:

[0104] Based on the captured video, a sparse point cloud containing three-dimensional position information is calculated;

[0105] Initialize the sparse point cloud to get a 3D Gaussian sphere, and reconstruct the 3DGS head based on the 3D Gaussian sphere;

[0106] Initialize the 3D Gaussian sphere position to obtain the human head point cloud;

[0107] A new 3D Gaussian sphere is expanded outside the human head point cloud. Through a predefined loop scheduling program, the offset δ of the new 3D Gaussian sphere is set based on the 3D Gaussian sphere position of the human head point cloud, and a shell-like structure is constructed to obtain the human head hair point cloud.

[0108] During 3DGS training, the 3D Gaussian sphere will continue to grow or shrink. When the task of head reconstruction is heavy, we will put the 3D Gaussian sphere on the head model during initialization, such as the surface of 3DMM. We then hope that the 3D Gaussian sphere can be generated outside the head to reconstruct the hair, but we do not want the Gaussian sphere to grow inside the head, because the latter is meaningless.

[0109] The present embodiment provides a Gaussian sphere updating method for improving the efficiency of single-video 3DGS head reconstruction, which is to solve the problem of effectively generating a Gaussian sphere in a directional outward direction, expand a new 3D Gaussian sphere outside the point cloud of the human head skull, and set the offset δ of the new 3D Gaussian sphere based on the 3D Gaussian sphere position of the human head skull point cloud through a predefined loop scheduling program to construct a shell-like structure and obtain a point cloud of human head hair. Therefore, the present embodiment provides a Gaussian sphere updating method for improving the efficiency of single-video 3DGS head reconstruction, which achieves the effect of spreading the 3D Gaussian sphere outward along the hair, and does not waste computing power in areas without hair.

[0110] Furthermore, in a Gaussian sphere updating method for improving single-video 3DGS head reconstruction of this embodiment, a new 3D Gaussian sphere is expanded outside the head point cloud, and an offset δ of the new 3D Gaussian sphere is set based on the 3D Gaussian sphere position of the head point cloud through a predefined loop scheduling program to construct a shell-like structure, and the head hair point cloud is obtained, including:

[0111] p(i) represents the position j of the new 3D Gaussian sphere on the i-th triangle patch, and δ(i) is the offset assigned to the new 3D Gaussian sphere. The growth formula of the new 3D Gaussian sphere is:

[0112]

[0113] in, is the 3D Gaussian ball growth formula of the human head point cloud, n(i) is the expansion direction parameter from the i-th triangle patch, It is the sampling point Coordinates on the i-th triangle patch; are the coordinates of three points on the i-th triangle patch.

[0114] Since the 3D Gaussian sphere is pruned and densified during the 3DGS model training process, we can eventually form a shell-like structure in a progressive and unsupervised manner, which can effectively capture fine-scale details.

[0115] As an optional implementation of this embodiment, in a Gaussian sphere update method for improving single-video 3DGS head reconstruction of this embodiment, the n(i) is the normal vector from the vertex of the triangle patch whose outward expansion direction parameter is the i-th triangle patch.

[0116] As an optional implementation of this embodiment, in a Gaussian sphere update method for improving single-video 3DGS head reconstruction of this embodiment, the n(i) is the midline of the triangle patch whose outward expansion direction parameter is the i-th triangle patch.

[0117] As an optional implementation of this embodiment, in a Gaussian sphere update method for improving single-video 3DGS head reconstruction of this embodiment, the n(i) is a perpendicular line from the i-th triangle patch whose outward expansion direction parameter is a triangle patch.

[0118] Specifically, in a Gaussian sphere updating method for improving single-video 3DGS head reconstruction of this embodiment, the offset δ of setting a new 3D Gaussian sphere based on the 3D Gaussian sphere position of the head point cloud includes:

[0119] The value range of the preset offset δ is [0, δmax], where δmax is the preset maximum offset;

[0120] Based on the 3D Gaussian sphere position of the human head point cloud, the new 3D Gaussian sphere offset δ is set in the value range [0, δmax].

[0121] This embodiment ensures that the 3D Gaussian sphere is expanded within the numerical range [0, δmax] by presetting the maximum offset δmax, constructs a shell-like structure, obtains the point cloud of the hair on the human head, and avoids wasting computing power in areas without hair.

[0122] As an optional implementation of this embodiment, a Gaussian sphere update method for improving the efficiency of single-video 3DGS head reconstruction in this embodiment, initializing the 3D Gaussian sphere position to obtain a human head skull point cloud includes: initializing the 3D Gaussian sphere position and placing it on the surface of the 3DMM head model to obtain a human head skull point cloud.

[0123] Experiments on eight open-source human heads on the market show that the new 3D Gaussian sphere growth formula of this embodiment can reduce the 3D Gaussian sphere calculation by half compared with the random point scattering method.

[0124] In terms of effect, see Figure 6 As shown in the comparison table, the "w / oδ" column is the experimental result data of the new 3D Gaussian sphere growth formula of this embodiment. The accurate calculation of the 3D Gaussian sphere position can improve the reconstruction effect. For specific visualization effects, see Figure 7 shown.

[0125] Embodiment 3

[0126] See also Figure 8 As shown, this embodiment provides a method for improving the efficiency of single-video 3DGS head reconstruction, including a method for reconstructing a 360-degree head:

[0127] The monocular video is used as input to the 3D perception GAN model for GAN inversion to obtain the side image data and back image data of 360-degree head reconstruction;

[0128] The side image data and back image data are used as pseudo data, and the monocular video is used as real data to input into the 3DGS model for model training to generate a 360-degree human head model.

[0129] In this embodiment, a method for reconstructing a 360-degree human head through 3DGS based on a monocular video is used. The side image data and the back image data are obtained by a generation method, and are absorbed as pseudo data to participate in the model training of real data through a training method to generate a 360-degree human head model.

[0130] Therefore, the method for reconstructing a 360-degree human head through 3DGS based on a monocular video of this embodiment can generate a 3D reconstruction result of the whole human head from a single-view video, and can cross-drive various motion amplitudes. The method for reconstructing a 360-degree human head through 3DGS based on a monocular video of this embodiment is the first solution that proposes to use a 3DGS solution to reconstruct 360° data using a single-view video.

[0131] Full head reconstruction: To reconstruct a 360° renderable head image from a monocular video with only the front view, a more powerful prior model must be used. We introduced PanoHead as a prior model, which is a 3D-aware GAN model that contains a lot of prior information about the side view and back view of the head. It mainly uses GAN inversion technology, and PTI is used for GAN inversion because it does not rely on additional encoder training and works well in the inversion of PanoHead.

[0132] GAN inversion refers to the process of mapping the target image into the latent space vector of the pre-trained GAN model, and then inputting the vector into the pre-trained generator to complete the image reconstruction. This process enables the pre-trained GAN model to be applied to real image editing and has strong interpretability.

[0133] Pivotal Tuning Inversion (PTI) is a technique for mapping real images to the latent space of StyleGAN, allowing for advanced editing. By fine-tuning the latent space of the generative model, PTI enables editing of real images, maintaining the accuracy of their identity features while reducing distortion.

[0134] Through experiments, we find that PanoHead performs well in generating static FFHQ aligned head images with small facial expression variations. In addition, we can also obtain head images with high-quality frontal rendering after the monocular reconstruction stage.

[0135] Therefore, we use images rendered with neutral expressions and poses as input for our analysis of avatars.

[0136] As an optional implementation of this embodiment, in a method of reconstructing a 360-degree human head through 3DGS based on a monocular video in this embodiment, before the monocular video is used as input and input into the 3D perception GAN model, the method includes:

[0137] The monocular video is input into a low-level vision pre-trained model for preprocessing, which injects more image-specific details into the wild input image of the monocular video to obtain high-quality input data.

[0138] Furthermore, in a method for reconstructing a 360-degree human head through 3DGS based on a monocular video in this embodiment, the low-level visual pre-training model is pre-trained based on a high-quality face dataset.

[0139] Face Restoration: The input images used for inversion are essentially in-the-wild images, which have a large domain gap from the high-quality dataset used for PanoHead training. Directly using rendered images for inversion, the optimized latent codes tend to break away from the GAN manifold, resulting in suboptimal images. The classic approach to address this problem is to apply domain adaptation or transfer learning. However, we can achieve the same goal more concisely. By processing the input images with a low-level vision model pre-trained on a high-quality face dataset, more image-specific details can be injected into the in-the-wild input images. This narrows the domain gap. We call this pre-trained model MR. In our experiments, we use the state-of-the-art face restoration model GFPGAN. It contains a generative prior on the FFHQ dataset that meets our needs.

[0140] As an optional implementation of this embodiment, a method for reconstructing a 360-degree human head through 3DGS based on a monocular video in this embodiment, wherein the 3D perception GAN model uses the Pivotal Tuning Inversion technology to perform GAN inversion;

[0141] Neutral face images of different angles are rendered with arbitrary camera postures, and neutral face images of different angles are filtered out through landmark detection to reconstruct good neutral face images. The filtered neutral face images are used to extend the Pivotal Tuning Inversion technology to multi-view situations.

[0142] Standard PTI takes as input a single image, then searches for latent codes and fine-tunes generator parameters. However, this approach tends to overfit the input image. In PanoHead, the performance of this naive approach drops dramatically in new views. It is worth noting that we can render neutral faces in arbitrary camera poses, and landmark detection allows us to filter out images with well-reconstructed faces. Therefore, we can use the filtered images to extend PTI to the multi-view case.

[0143] Specifically, in a method of reconstructing a 360-degree human head through 3DGS based on a monocular video in this embodiment, the use of a filtered neutral face image to extend the Pivotal Tuning Inversion technology to a multi-view situation includes:

[0144] Search key potential code wp:

[0145]

[0146] M is the number of valid multi-view graphs, L LPIPS is the perceived loss, is the face image after face restoration, G Pano is the frozen full head generator, θ is the generator parameter, and c is the camera pose;

[0147] Adjust the parameters of the full head generator, and the loss setting is as follows

[0148]

[0149] θ* is the optimized parameter, initialized using the pre-trained model θ.

[0150] After adding the side image data and the back image data, we can use this additional data to reconstruct the human head, but training directly on these data will lead to the degradation of the frontal perspective, so we developed a training strategy as follows.

[0151] In a method for reconstructing a 360-degree human head through 3DGS based on a monocular video of this embodiment, the side image data and the back image data are used as pseudo data, and the monocular video is used as real data to input into the 3DGS model for model training to generate a 360-degree human head model, which includes:

[0152] Sampling from real datasets

[0153] pass and c i Rendering

[0154] pass And back propagation calculates the loss L1;

[0155] Update the 3D point cloud {G};

[0156] Where, I = {Ii} video frame sequence;

[0157] Ψ={Ψi} tracked task expression;

[0158] Θ={Θi}, character posture;

[0159] c i It's the camera posture.

[0160] In a method for reconstructing a 360-degree human head through 3DGS based on a monocular video of this embodiment, the side image data and the back image data are used as pseudo data, and the monocular video is used as real data to input into the 3DGS model for model training to generate a 360-degree human head model, which includes:

[0161] Sampling from a pseudo dataset Replace with Ψ i ;

[0162] pass and c j Rendering

[0163] pass And back propagation calculates the loss L2;

[0164] Update the 3D point cloud {G}.

[0165] A method of reconstructing a 360-degree human head through 3DGS based on a monocular video in this embodiment cross-trains between pseudo data and real data, similar to the training of an adversarial network. Because the Gaussian sphere behind the human head will be cropped during the training process, an additional Gaussian sphere set is initialized on the basis of the existing Gaussian field to model the back of the human head. Although each pseudo data image is generated using a neural network expression, we observe that most tables do not affect the appearance of the side and back. Therefore, when training pseudo data, expression Ψi is used instead of an all-zero vector.

[0166] Experiments show that the training strategy of this embodiment improves the effect on the sides and back of the head.

[0167] Specifically, in this embodiment, a method for reconstructing a 360-degree human head through 3DGS based on a monocular video is used. For example, the training strategy adopted is Fig. 9 shown.

[0168] The effect of reconstructing a 360-degree human head through 3DGS based on a monocular video in this embodiment is shown as follows:

[0169] See also Fig.10 and Fig.11 As shown in the figure, the reconstruction effect of the head from the side and back is shown. It can be seen that compared with the best solutions on the market, the solution of this embodiment (the "ours" column in the figure) is better. Both multi-view and additional auxiliary training bring obvious positive benefits.

[0170] See also Fig.12 and Fig.13 As shown, the effects of self-driving and cross-driving are demonstrated. The scheme of this embodiment ( Fig.12 The "ours" column and Fig.13 The rightmost column in the figure all work well in expressing details.

[0171] Embodiment 4

[0172] The following describes an electronic device embodiment of the present invention, which can be regarded as a specific physical implementation of the method and device embodiments of the present invention. The details described in the electronic device embodiment of the present invention should be regarded as a supplement to the above method or device embodiments; details not disclosed in the electronic device embodiment of the present invention can be implemented with reference to the above method or device embodiments.

[0173] Fig.14 It is a structural schematic diagram of an electronic device of an embodiment of the present invention, the electronic device includes a processor and a memory, the memory is used to store a computer executable program, when the computer program is executed by the processor, the processor executes a single video 3DGS head reconstruction method of embodiment one, or two, or three.

[0174] like Fig.14 As shown, the electronic device is presented in the form of a general computing device. The processor may be one or more and work in coordination. The present invention does not exclude distributed processing, that is, the processor may be dispersed in different physical devices. The electronic device of the present invention is not limited to a single entity, but may also be the sum of multiple physical devices.

[0175] The memory stores a computer executable program, which is usually a machine-readable code. The computer-readable program can be executed by the processor to enable the electronic device to perform the method of the present invention, or at least part of the steps in the method.

[0176] The memory includes a volatile memory, such as a random access memory unit (RAM) and / or a cache memory unit, and may also be a non-volatile memory, such as a read-only memory unit (ROM).

[0177] Optionally, in this embodiment, the electronic device further includes an I / O interface, which is used for the electronic device to exchange data with an external device. The I / O interface can represent one or more of several types of bus structures, including a storage unit bus or a storage unit controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of a variety of bus structures.

[0178] It should be understood that Fig.14 The electronic device shown is only an example of the present invention, and the electronic device of the present invention may also include elements or components not shown in the above examples. For example, some electronic devices also include display units such as display screens, and some electronic devices also include human-computer interaction elements such as buttons, keyboards, etc. As long as the electronic device can execute the computer-readable program in the memory to implement the method of the present invention or at least part of the steps of the method, it can be considered as an electronic device covered by the present invention.

[0179] Fig.15 Schematic diagram of a computer readable recording medium according to an embodiment of the present invention. Fig.15 As shown, a computer executable program is stored in a computer-readable recording medium, and when the computer executable program is executed, a single-video 3DGS head reconstruction method of Embodiment 1, or 2, or 3 of the present invention is implemented. The computer-readable recording medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The readable recording medium may also be any readable medium other than a readable recording medium, which may send, propagate, or transmit a program for use by or in combination with an instruction execution system, device, or device. The program code contained on the readable recording medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the above.

[0180] Program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0181] Through the above description of the implementation mode, it is easy for those skilled in the art to understand that the present invention can be implemented by hardware capable of executing a specific computer program, such as the system of the present invention, and the electronic processing unit, server, client, mobile phone, control unit, processor, etc. contained in the system. The present invention can also be implemented by computer software that executes the method of the present invention, such as control software executed by a microprocessor, an electronic control unit, a client, a server, etc. However, it should be noted that the computer software that executes the method of the present invention is not limited to being executed by one or a specific hardware entity, and it can also be implemented in a distributed manner by unspecified specific hardware. For computer software, the software product can be stored in a computer-readable recording medium (which can be a CD-ROM, a USB flash drive, a mobile disk, etc.), and can also be distributed and stored on the network, as long as it enables the electronic device to execute the method according to the present invention.

[0182] The above embodiments are only used to illustrate the present invention and are not intended to limit the technical solutions described in the present invention. Although the present invention has been described in detail with reference to the above embodiments, the present invention is not limited to the above specific implementation methods. Therefore, any modification or equivalent replacement of the present invention; and all technical solutions and improvements thereof that do not depart from the spirit and scope of the invention are included in the scope of the claims of the present invention.

Claims

1. A densification method for single video 3DGS head reconstruction, characterized in that: include: Based on the video samples, a sparse point cloud containing three-dimensional position information is calculated; Initialize the sparse point cloud to get a 3D Gaussian sphere; The 3DGS model is trained based on the 3D Gaussian sphere. In the adaptive densification control process of the 3DGS model training, the densification index is introduced into the sampling probability function. The best densification path is automatically updated during the 3DGS model training process, and the 3DGS model is obtained after the training is completed.

2. The method for densification of single video 3DGS head reconstruction according to claim 1, characterized in that: The sampling probability function is: in, is the importance index of a certain 3D Gaussian sphere, and the denominator represents the sum of the importance indexes of all 3D Gaussian spheres of the triangle face where the certain 3D Gaussian sphere is located.

3. The method for densification of single video 3DGS head reconstruction according to claim 1, characterized in that: The importance indexes of the sampling probability function include: maximum index max(s), maximum value to minimum value index max(s) / min(s), parameter gradient index, and transparency index; wherein s represents position information, i.e., the gradient relative to the position coordinates (x, y, z) of the 3D Gaussian sphere.

4. The method for densification of single video 3DGS head reconstruction according to claim 1, characterized in that: The sampling probability function adds different weights to different designated important areas of the human head reconstruction.

5. The method for densification of single video 3DGS head reconstruction according to claim 1, characterized in that: The 3DGS model training based on the 3D Gaussian sphere includes: For every N iterations of the 3D Gaussian sphere, the order of the spherical harmonic coefficients is increased, where N is the set value; Randomly select a camera perspective; Render the image, obtaining the view space point, visibility filter, and radius information; Calculate the loss L = (1-λ)L1 + λL D-SSIM (L1 loss and L D-SSIM The weighted sum of the losses, L1 loss helps ensure pixel-level accuracy, and L D-SSIM The loss helps to maintain the overall structure and visual quality of the image), λ is the preset weight value, and back propagation is performed; Subsequent operations via gradient-free contextual learning include: Perform point cloud density operations based on the number of iterations: Update the maximum radius information; Increase and trim point cloud density based on conditions; Update the optimizer parameters; During the point cloud density operation, the densification index is introduced into the sampling probability function, which is: in, is the importance index of a certain 3D Gaussian sphere, and the denominator represents the sum of the importance indexes of all 3D Gaussian spheres of the triangle face where the certain 3D Gaussian sphere is located.

6. The method for densification of single video 3DGS head reconstruction according to claim 1, characterized in that: include: Shoot an image or video of the target person, input the image or video into the trained 3DGS model, and generate a 3D reconstruction model of the target person.

7. A densification device for single video 3DGS head reconstruction, characterized in that: It includes a 3DGS model training module, and the model training module includes: The point cloud acquisition unit calculates and obtains a sparse point cloud covering three-dimensional position information based on the video samples; 3D Gaussian sphere generation unit, which initializes the sparse point cloud to obtain a 3D Gaussian sphere; The 3DGS model training unit performs 3DGS model training based on the 3D Gaussian sphere. In the adaptive densification control process of the 3DGS model training, the densification index is introduced into the sampling probability function. The best densification path is automatically updated during the 3DGS model training process, and the 3DGS model is obtained after the training is completed.

8. The densification device for single video 3DGS head reconstruction according to claim 7, characterized in that: It includes a three-dimensional reconstruction module: obtaining the image video of the target person, inputting the image video into the 3DGS model, and generating a three-dimensional reconstruction model of the target person.

9. An electronic device comprising a processor and a memory, wherein the memory is used to store a computer executable program, characterized in that: When the computer program is executed by the processor, the processor executes the densification method for single-video 3DGS head reconstruction as described in any one of claims 1-6.

10. A computer-readable recording medium storing a computer-executable program, characterized in that: When the computer executable program is executed, a densification method for single video 3DGS head reconstruction as described in any one of claims 1 to 6 is implemented.

Images