Digital human reconstruction method and system based on grid and Gaussian point mixed representation

Abstract

The invention discloses a digital human reconstruction method and system based on grid and Gaussian point mixed representation, and the method comprises the steps: carrying out the analysis and processing of a human body image in a video data set, and obtaining a human body grid and posture parameters; subdividing the human body grid, binding a vertex of the refined grid with a three-dimensional Gaussian point, and initializing a standard space Gaussian position; performing feature coding on the Gaussian position, and learning geometric attributes and color attributes of Gaussian points in combination with the feature codes and the attitude parameters to obtain a complete Gaussian attribute set; and according to the complete Gaussian attribute set, generating a digital human image by adopting a linear hybrid skin method in combination with optimization constraints. According to the invention, stable reconstruction and high-quality rendering of different character images under different postures can be realized.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a method and system for digital human reconstruction based on a mixture of grid and Gaussian point representations. Background Technology

[0002] With the rapid development of virtual reality (VR), augmented reality (AR), gaming, and digital human technologies, reconstructing high-fidelity and animable 3D human models from monocular video has become a crucial technological direction in computer vision and computer graphics. This technology not only requires models to accurately capture the subtle geometric structures and appearance features of the human body, but also to maintain consistency between body structure and identity characteristics under different postures and facial expressions, thereby supporting high-quality dynamic rendering and interactive applications. This type of technology has wide-ranging applications in virtual character creation, digital human interaction, and film and television production.

[0003] Existing parametric human models, such as SMPL and SMPL-X, have made significant progress in human geometry and pose modeling. Their explicit mesh structures provide clear topological relationships and semantic consistency, offering strong priors for animation control, reorientation, and downstream editing tasks. However, these mesh-based explicit methods are limited by fixed topology and finite resolution, making it difficult to accurately capture facial expressions, clothing dynamics, and high-frequency texture details, thus limiting their performance in complex, non-rigid deformation scenes. In contrast, implicit neural representations (such as NeRF) can model geometric and appearance features in continuous space, exhibiting higher realism and rendering quality. NeRF-based human reconstruction methods learn 3D shapes and textures from high-quality videos through implicit volume functions, resulting in high-fidelity dynamic appearances. However, these methods typically require long training times and high computational costs, resulting in relatively slow rendering speeds and unsuitability for real-time interaction. Furthermore, implicit representations lack explicit topological structures, making it difficult to perform semantic-based animation and editing operations. In addition, point-based methods have also demonstrated strong modeling capabilities. These methods achieve efficient rendering and rapid optimization by explicitly representing local voxel or point structures in space. However, due to the lack of topological constraints between points, such methods are prone to geometric drift and texture artifacts under pose changes or large deformations, leading to structural instability, especially in dynamic scenes with complex facial expressions and non-rigid clothing. To address these issues, recent research has proposed a hybrid human representation method combining parametric meshes and 3D Gaussians. Gaussian points are bound to the surface of the human body mesh, allowing it to deform with mesh vertices or faces. This method can capture subtle dynamic geometric and texture changes while maintaining the stability of the underlying mesh. Hybrid representation combines the explicit topological consistency of the mesh with the appearance expressiveness of implicit methods, supporting fast training and real-time rendering, and demonstrating superior performance in monocular video learning. However, existing hybrid methods still have limitations. Their underlying canonical meshes are usually derived from standardized human body models, without directly adjusting the mesh geometry. They only indirectly fit the appearance by modifying the position, scale, or color of Gaussian points, limiting the model's ability to express the geometry of non-standard body shapes or complex local structures. This makes it difficult to accurately reconstruct high-frequency geometric features such as hands, faces, and clothing wrinkles, and they are prone to geometric drift and structural instability under large pose changes or non-rigid deformations. Summary of the Invention

[0004] The purpose of this invention is to provide a digital human reconstruction method and system based on a hybrid representation of meshes and Gaussian points, which solves the problems of insufficient geometric controllability, poor structural stability, and insufficient fitting ability for complex non-rigid deformations in existing digital human reconstruction technologies. By binding three-dimensional Gaussian points with SMPL-X mesh vertices, a posture-related deformation and appearance modeling mechanism is introduced, and the depth information and normal information of the character are combined to perform geometric supervision on the digital human reconstruction process, so as to achieve stable reconstruction and high-quality rendering of different character images under different postures.

[0005] To achieve the above objectives, the present invention provides the following solution:

[0006] Digital human reconstruction methods based on a mixture of grid and Gaussian point representations include:

[0007] Analyze and process human images in the video dataset to obtain human mesh and pose parameters;

[0008] The human body mesh is subdivided, and the vertices of the refined mesh are bound to three-dimensional Gaussian points to initialize the normalized space Gaussian positions;

[0009] The Gaussian position is feature-encoded, and the geometric and color attributes of the Gaussian point are learned by combining the feature encoding and the pose parameters to obtain a complete set of Gaussian attributes.

[0010] Based on the complete set of Gaussian attributes, a linear blending skinning method combined with optimization constraints is used to generate digital human images.

[0011] Optionally, the human images in the video dataset are analyzed and processed to obtain human mesh and pose parameters, including:

[0012] Extract key points, pose parameters, and inter-frame position information of the person from the video dataset, and generate depth maps and normal maps as geometric supervision information to obtain the human body mesh and pose parameters.

[0013] Optionally, subdividing the human body mesh and binding the vertices of the refined mesh to three-dimensional Gaussian points includes: subdividing the human body mesh using PyTorch3D, binding the vertices of the refined mesh to three-dimensional Gaussian points one-to-one, and using the three-dimensional positions of the vertices of the refined mesh as the initial values ​​of the Gaussian point positions.

[0014] Optionally, feature encoding of the Gaussian positions includes:

[0015] A three-plane feature encoding network is used to encode the Gaussian positions, and high-resolution three-plane features are introduced into the facial region.

[0016] Optionally, learning the geometric and color properties of Gaussian points includes:

[0017] The feature encoding is performed by learning the geometric and appearance attributes of each Gaussian point through the geometric and appearance branches of the dual-branch neural network structure, respectively. The geometric and appearance branches are composed of two multilayer perceptrons (MLPs). The basic MLP is used to learn static identity features, and the pose-related MLP models non-rigid deformation by input pose parameters.

[0018] Optionally, the optimization constraints include: color consistency constraints, regularization constraints, and geometric consistency constraints;

[0019] Wherein, the regularization constraint adopts Norm regularization is used to constrain the Gaussian position and scale offsets predicted by the geometric network;

[0020] The geometric consistency constraints include consistency constraints based on normal maps and local and global consistency constraints based on depth maps.

[0021] This invention also provides a digital human reconstruction system based on a hybrid representation of grids and Gaussian points, comprising:

[0022] The mesh acquisition module is used to analyze and process human images in video datasets to obtain human mesh and pose parameters;

[0023] The Gaussian binding module is used to subdivide the human body mesh, bind the vertices of the refined mesh to three-dimensional Gaussian points, and initialize the Gaussian positions in the normalized space.

[0024] The feature encoding module is used to perform feature encoding on the Gaussian positions;

[0025] The Gaussian attribute learning module is used to combine the feature encoding and pose parameters to learn the geometric and color attributes of Gaussian points and obtain a complete set of Gaussian attributes.

[0026] The digital human construction module is used to generate digital human images based on the complete set of Gaussian attributes, using a linear blending skinning method combined with optimization constraints.

[0027] Optionally, the Gaussian binding module includes: mesh subdivision units and Gaussian binding units;

[0028] The mesh subdivision unit is used to subdivide the human body mesh using PyTorch3D;

[0029] The Gaussian binding unit is used to bind the vertices of the refined mesh to three-dimensional Gaussian points in a one-to-one correspondence, and to use the three-dimensional position of the vertex of the refined mesh as the initial value of the position of the Gaussian point.

[0030] Optionally, the Gaussian attribute learning module includes: a geometric network unit and a color network unit;

[0031] The geometric network unit is used to learn the geometrically relevant attributes of each Gaussian point through the geometric network branches of the dual-branch neural network structure for the feature encoding;

[0032] The color network unit is used to learn the appearance attributes of each Gaussian point through the appearance network branches of the dual-branch neural network structure.

[0033] Optionally, the digital human building module includes: a pose-driven unit, a Gaussian rendering unit, a color consistency unit, a regularization constraint unit, and a geometric consistency unit;

[0034] The posture driving unit is used to transform the complete set of Gaussian attributes according to the posture parameters of the character in the current frame through a linear blending skinning method, thereby realizing posture driving.

[0035] The Gaussian rendering unit is used to render the complete Gaussian properties using a three-dimensional Gaussian sputtering method to generate a high-quality digital human image.

[0036] The color consistency unit is used to constrain the color consistency between the rendered image and the real image using RGB, SSIM, and LPIPS metrics.

[0037] The regularization constraint unit is used to employ Norm regularization is used to constrain the Gaussian position and scale offsets predicted by the geometric network;

[0038] The geometric consistency unit is used to implement consistency constraints based on normal maps and local and global consistency constraints based on depth maps.

[0039] The beneficial effects of the present invention are: (1) Enhanced detail fidelity: The synergistic effect of pose-related MLP and geometric supervision effectively restores high-frequency details such as clothing, facial expressions and fingers;

[0040] (2) Improved geometric accuracy: The introduction of depth and normal consistency constraints makes the reconstructed surface smoother and the structure more continuous;

[0041] (3) Improved pose generalization: Based on the Gaussian binding and linear hybrid skinning strategy in normal space, the model can adapt to diverse body shapes and complex movements. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 This is a structural diagram of a digital human reconstruction system based on a mixture of grid and Gaussian point representation according to an embodiment of the present invention;

[0044] Figure 2 This is a framework diagram of the Gaussian attribute learning module according to an embodiment of the present invention;

[0045] Figure 3 This is a flowchart of a digital human reconstruction method based on a mixture of grid and Gaussian point representation according to an embodiment of the present invention;

[0046] Figure 4 This is a schematic diagram illustrating the experimental results of the digital human reconstruction method based on a mixture of grid and Gaussian point representation in an embodiment of the present invention on a publicly available human dataset.

[0047] Figure 5 This is a schematic diagram of multiple reconstruction results of the digital human reconstruction method based on a mixture of grid and Gaussian point representation in an embodiment of the present invention. Detailed Implementation

[0048] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0049] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0050] like Figure 3 As shown, this embodiment proposes a digital human reconstruction method based on a hybrid representation of grids and Gaussian points, including:

[0051] Analyze and process human images in the video dataset to obtain human mesh and pose parameters;

[0052] The human body mesh is subdivided, and the vertices of the refined mesh are bound to 3D Gaussian points to initialize the Gaussian positions in the normalized space.

[0053] The Gaussian position is feature-encoded, and the geometric and color attributes of the Gaussian point are learned by combining the feature encoding and the pose parameters to obtain a complete set of Gaussian attributes.

[0054] Based on the complete set of Gaussian properties, a linear blending skinning method combined with optimization constraints is used to generate digital human images.

[0055] Furthermore, the human images in the video dataset are analyzed and processed to obtain human mesh and pose parameters, including:

[0056] Extract key points, pose parameters, and inter-frame position information of people from the video dataset, and generate depth maps and normal maps as geometric supervision information to obtain human body mesh and pose parameters.

[0057] Specifically, the human images in the video dataset undergo data preprocessing, including human parameter extraction and supervised information generation, to obtain the initial geometric information, usable mask data, depth information, and normal information of the human body for each frame. Then, a set of anisotropic 3D Gaussian points is used as the explicit spatial representation, with each Gaussian point centered on its mean position vector μ∈ℝ³, and represented by the covariance matrix. Its spatial distribution is defined to satisfy the following relationship:

[0058] ;

[0059] Wherein, the covariance matrix It is a positive semi-definite matrix, which can be decomposed into a combination of the scaling matrix S and the rotation matrix R. To facilitate parameterized storage, the scale matrix S is represented as a three-dimensional diagonal vector. The rotation matrix R is represented by quaternions. Encode the data. The complete properties of each Gaussian point are defined as follows: ,in, Indicates the center position of Gauss. Indicates rotation parameters, Represents the scale vector. This represents the opacity parameter. This indicates color or appearance information. It enhances individualized modeling capabilities.

[0060] Furthermore, the human body mesh is further subdivided, and the vertices of the refined mesh are bound to three-dimensional Gaussian points. This includes: subdividing the human body mesh using methods provided by PyTorch3D, binding the vertices of the refined mesh to three-dimensional Gaussian points one-to-one, and using the three-dimensional positions of the vertices of the refined mesh as the initial values ​​of the Gaussian point positions.

[0061] Specifically, in this embodiment, the mesh is subdivided based on the standard SMPL-X model, and each vertex is... The three-dimensional position as the center of Gauss The initial values ​​establish a one-to-one correspondence between the grid and the Gaussian points. Binding the Gaussian points to the grid means using the spatial position corresponding to the grid vertex as the initial spatial position of the Gaussian point.

[0062] Furthermore, feature encoding of Gaussian positions includes:

[0063] A three-plane feature encoding network is used to encode Gaussian locations, and high-resolution three-plane features are introduced into the facial region.

[0064] Specifically, a three-plane feature encoding network is used to encode the normalized Gaussian positions and extract the spatial feature vector of the human body. High-resolution three-plane features are introduced in the facial region to improve the ability to express details. The normalized Gaussian position is the position (x, y, z) of the vertex of the SMPL-X mesh in space.

[0065] Furthermore, learning the geometric and color properties of Gaussian points includes:

[0066] The feature encoding is performed by learning the geometric and appearance attributes of each Gaussian point through the geometric and appearance branches of the dual-branch neural network structure, respectively. The geometric and appearance branches are composed of two multilayer perceptrons (MLPs). The basic MLP is used to learn static identity features, and the pose-related MLP models non-rigid deformation by input pose parameters.

[0067] Specifically, the feature encoding is fed into a dual-branch neural network structure to learn Gaussian properties, including a geometric network branch and an appearance network branch, which respectively learn the geometrically related properties (position) of each Gaussian point. ,scale ) and appearance attributes (color) Learn, rotate Set to identity matrix, opacity Set to 1. Each branch consists of two multilayer perceptrons (MLPs): a base MLP for learning static identity features such as body geometry and basic appearance, and a pose-related MLP for modeling non-rigid deformations such as clothing wrinkles and facial expression changes through input pose parameters. Finally, by fusing the base attributes and pose-related offsets, a complete set of Gaussian attributes is obtained, achieving a normalized geometric and appearance representation of the human body model.

[0068] In the animation stage, this embodiment uses a linear blending skinning method to transform Gaussian points in the normalized space according to pose parameters, thereby achieving pose-driven digital human animation generation. In the rendering stage, 3D Gaussian sputtering is employed to achieve fast, high-quality image generation.

[0069] Furthermore, the optimization constraints include: color consistency constraints, regularization constraints, and geometric consistency constraints;

[0070] Among them, regularization constraints are adopted. Norm regularization is used to constrain the Gaussian position and scale offsets predicted by the geometric network;

[0071] Geometric consistency constraints include consistency constraints based on normal maps and local and global consistency constraints based on depth maps. Specifically, to improve the accuracy and stability of reconstruction, this embodiment designs several optimization constraints, including color reconstruction loss, regularization loss, and geometric constraint loss. Color consistency loss constrains the consistency between the rendered result and the real image at the pixel, structural, and perceptual levels using RGB, SSIM, and LPIPS metrics; regularization loss constrains the range of Gaussian property variations and local smoothness, ensuring deformation continuity through Laplacian regularization; geometric consistency loss, based on depth map and normal map supervision, improves the accuracy and stability of human geometric structures from the perspectives of spatial consistency and surface smoothness, respectively.

[0072] Through the above design, this embodiment achieves high-precision geometric reconstruction and appearance modeling of the human body under different postures and expressions. It has high structural controllability and geometric stability, and solves the problems of insufficient non-rigid deformation fitting ability, geometric instability and limited detail restoration of existing Gaussian digital human reconstruction methods. It can be widely used in virtual reality, augmented reality, film and television production, game development and digital human interaction.

[0073] like Figure 1-2 As shown, this embodiment also provides a digital human reconstruction system based on a hybrid representation of grids and Gaussian points, including...

[0074] The mesh acquisition module is used to analyze and process human images in video datasets to obtain human mesh and pose parameters;

[0075] The Gaussian binding module is used to subdivide the human body mesh, bind the vertices of the refined mesh to 3D Gaussian points, and initialize the Gaussian positions in the normal space.

[0076] The feature encoding module is used to encode the features of Gaussian positions;

[0077] The Gaussian attribute learning module is used to combine feature encoding and pose parameters to learn the geometric and color attributes of Gaussian points and obtain a complete set of Gaussian attributes.

[0078] The digital human construction module is used to generate digital human images based on a complete set of Gaussian attributes, using a linear blending skinning method combined with optimization constraints.

[0079] Furthermore, the grid acquisition module includes a human parameter extraction unit and a supervision information generation unit;

[0080] The human body parameter extraction unit is used to extract human body parameters from the input monocular video or image sequence, including human keypoint detection and pose parameter extraction. This unit generates initial geometric information, skeleton parameters, and mask data for each frame of the human body, providing basic input for subsequent mesh binding and Gaussian attribute learning.

[0081] The supervision information generation unit is used to generate geometric supervision information for video frames using the Sapiens method, including depth maps and normal maps. The depth map is used to represent the spatial distance information of each pixel, and the normal map is used to provide surface orientation information, thereby guiding subsequent Gaussian property learning and geometric optimization.

[0082] Furthermore, the Gaussian binding module includes: mesh subdivision cells and Gaussian binding cells;

[0083] Mesh subdivision unit, used to subdivide the human body mesh using methods provided by PyTorch3D;

[0084] Specifically, the SMPL-X mesh is upsampled and subdivided using the subdivision methods provided by PyTorch3D to increase the number of mesh vertices, thereby enabling more precise binding of Gaussian points and capture of high-frequency geometric features of the human body and clothing. The number of subdivisions can be flexibly adjusted according to specific needs.

[0085] Gaussian binding cells are used to bind the vertices of the refined mesh to the 3D positions of the Gaussian points, using the 3D positions of the vertices of the refined mesh as the initial values.

[0086] Specifically, 3D Gaussian points are bound to the vertices of the subdivided mesh, and Gaussian positions in the normalized space are initialized, providing a basic geometric framework for the hybrid 3D representation of the human body. The Gaussian positions are denoted as the base positions. This serves as an initial reference for subsequent learning of geometric and appearance attributes.

[0087] Furthermore, the feature encoding module includes a feature encoding unit that employs a three-plane feature encoding method. This method maps Gaussian positions in normal space to three projection planes in three-dimensional space, extracting the spatial feature vector for each Gaussian point. High-resolution three-plane features are introduced into the facial region to enhance the ability to express facial details.

[0088] Furthermore, the Gaussian property learning module includes: geometric network units and color network units;

[0089] Geometric network units are used to learn the geometric properties of each Gaussian point through the geometric network branches of a two-branch neural network structure;

[0090] Specifically, it is used to learn geometrically relevant Gaussian properties, including positional offsets and scale adjustments. This unit contains two branch networks: a base network that captures static geometric features of the human body and learns positional offsets. and basic scale value Attitude-dependent networks are used to model non-rigid deformations based on input attitude parameters and learn attitude-related position offsets. Attitude-related scale offset Ultimately, the basic position. With position offset and attitude-related position offset Adding them together forms the complete Gaussian position property. Basic scale value Attitude-related scale offset Adding them together forms the complete Gaussian scale property. Used for subsequent animation driving and rendering, denoted as:

[0091] ;

[0092] .

[0093] The color network unit is used to learn the appearance attributes of each Gaussian point through the appearance network branches of the dual-branch neural network structure.

[0094] Specifically, it is used to learn Gaussian color attributes. This unit also contains two branch networks: a base network for learning the static appearance features of the human body and learning basic colors. Pose-dependent networks are used to adjust appearance features based on pose parameters, capture facial expression changes and dynamic textures, and learn pose-dependent color offsets. Basic colors Posture-related color offset Adding them together forms the complete Gaussian color attribute. Used for subsequent animation driving and rendering, denoted as:

[0095] .

[0096] Furthermore, the digital human building module includes: a pose-driven unit, a Gaussian rendering unit, a color consistency unit, a regularization constraint unit, and a geometric consistency unit;

[0097] The pose-driven unit is used to transform the complete set of Gaussian attributes according to the pose parameters of the character in the current frame through a linear blending skinning method, thereby achieving pose-driven behavior.

[0098] Specifically, by using the linear blending skinning method, Gaussian points in the normal space are transformed according to the pose parameters of the character in the current frame, thus realizing pose-driven digital human animation.

[0099] The Gaussian rendering unit is used to render the complete Gaussian properties using a 3D Gaussian sputtering method, generating high-quality digital human images and achieving fast and realistic visual presentation.

[0100] The color consistency unit is used to constrain the color consistency between the rendered image and the real image using RGB, SSIM, and LPIPS metrics.

[0101] Specifically, the color consistency between the rendered image and the real image is constrained by RGB, SSIM, and LPIPS metrics, ensuring that the rendered result highly matches the original data at the pixel, structural, and perceptual levels. , , Loss items , , The weighting coefficients then determine the color consistency loss. Recorded as:

[0102] .

[0103] Regularization constraint unit, used to adopt Norm regularization is used to constrain the Gaussian position and scale offsets predicted by the geometric network;

[0104] Specifically, constraints are imposed on the location and scale properties of Gaussian points to ensure their local continuity and deformation smoothness, thereby improving the stability of geometric reconstruction. This unit adopts... Norm regularization constrains the Gaussian position and scale offsets predicted by the geometric network, ensuring moderate variations in Gaussian properties and preventing excessive deviations from the initial position. Simultaneously, Laplacian smoothing regularization ensures the continuity of Gaussian properties in the local space, resulting in smoother deformation and reduced geometric abrupt changes. During the optimization phase, this element also regularizes joint position offsets to further stabilize the geometry of the human skeleton and the mesh. , , , The mean regularization loss for location and scale is calculated using the weighting coefficients of the loss term. Laplace regularization loss for position and scale Joint position offset regularization loss They are respectively recorded as:

[0105] ;

[0106] ;

[0107] ;

[0108] The total regularization loss is denoted as:

[0109] .

[0110] Geometric consistency elements are used to implement normal consistency constraints, as well as depth local consistency constraints and depth global consistency constraints.

[0111] Specifically, by generating corresponding normal maps and depth maps from the rendered frames, geometric constraints are applied to the 3D human model bound to Gaussian points to improve the model's geometric accuracy and spatial consistency. This unit first calculates vertex normals based on mesh faces and assigns them to the bound Gaussian points. After rendering, a normal map is generated for comparison with high-quality geometric labels. To enhance the constraint effect, this unit calculates the absolute error and directional difference of the normals as normal loss. Training is performed only within the effective pixel area to avoid background interference. In depth loss... In this unit, both local and global depth supervision are introduced to ensure the consistency of the model in local high-frequency structures and global spatial distribution. Local depth supervision divides the depth map into multiple small regions, normalizes each region, and calculates pixel-level depth consistency. Global depth supervision uses a uniform standard deviation to normalize pixels, ensuring that the depth distribution throughout the space is consistent with high-quality labels. Finally, by combining normal loss and depth loss to form geometric consistency loss, precise geometric guidance is provided for Gaussian attribute learning and mesh deformation, achieving high-precision 3D structure reconstruction and surface smoothness optimization of the digital human model. , As the weighting coefficients, the depth loss and normal loss are denoted as follows:

[0112] ;

[0113] ;

[0114] Total geometric consistency loss Recorded as:

[0115] ;

[0116] Final total loss Recorded as follows:

[0117] .

[0118] This embodiment significantly improves the accuracy, detail fidelity, and pose generalization ability of human body reconstruction by introducing a geometric constraint optimization mechanism and a Gaussian representation fusion strategy in the process of animable digital human modeling. Experimental results show that the technical solution of this invention achieves superior performance compared to existing methods on multiple public datasets, and demonstrates significant improvements in geometric consistency, detail restoration, pose stability, and animability. It can be widely applied in fields such as virtual human generation, film and television production, real-time interaction, and immersive digital content creation, yielding significant social and economic benefits. Figure 4 , Figure 5 As shown, the human image rendered by the method in this embodiment exhibits excellent visual quality and geometric consistency. The generated depth and normal information are highly accurate and smooth, realistically reflecting the geometric shape and spatial structure of the human surface.

[0119] On the one hand, the dual-branch design of geometric and color network units proposed in this embodiment can stably capture the non-rigid feature changes of the human body, achieving rich and geometrically coherent digital human reconstruction. Qualitative experimental results show that the Gaussian point cloud generated by this invention has a uniform density distribution and smooth surface transition, with no obvious misalignment or collapse at limb connections. Especially in the hand and facial regions, this invention can accurately reconstruct finger details and facial expression structures, ensuring natural proportions and surface continuity. On the other hand, thanks to the geometric supervision joint optimization mechanism, by introducing local and global consistency constraints in the normal and depth domains, the surface burrs and geometric collapse phenomena commonly found in traditional methods are effectively suppressed. In experiments on the X-Humans and Neuman public datasets, this invention achieves the best results in image reconstruction quality metrics PSNR, SplattingAvatar, and 3DGS-Avatar compared to existing methods such as GaussianAvatar, SplattingAvatar, and 3DGS-Avatar, with an overall performance improvement of approximately 10% to 15%. These results demonstrate that this invention has significant advantages in overall reconstruction accuracy, geometric consistency, and texture fidelity.

[0120] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made to the technical solutions of the present invention by those skilled in the art without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. A digital human reconstruction method based on a hybrid representation of grids and Gaussian points, characterized in that, include: Analyze and process human images in the video dataset to obtain human mesh and pose parameters; The human body mesh is subdivided, and the vertices of the refined mesh are bound to three-dimensional Gaussian points to initialize the normalized space Gaussian positions; The Gaussian position is feature-encoded, and the geometric and color attributes of the Gaussian point are learned by combining the feature encoding and the pose parameters to obtain a complete set of Gaussian attributes. Based on the complete set of Gaussian attributes, a linear blending skinning method combined with optimization constraints is used to generate digital human images.

2. The digital human reconstruction method based on a mixture of grid and Gaussian point representation according to claim 1, characterized in that, The human images in the video dataset are analyzed and processed to obtain human mesh and pose parameters, including: Extract key points, pose parameters, and inter-frame position information of the person from the video dataset, and generate depth maps and normal maps as geometric supervision information to obtain the human body mesh and pose parameters.

3. The digital human reconstruction method based on a mixture of grid and Gaussian point representation according to claim 1, characterized in that, Subdividing the human body mesh and binding the vertices of the refined mesh to three-dimensional Gaussian points includes: subdividing the human body mesh using PyTorch3D, binding the vertices of the refined mesh to three-dimensional Gaussian points one-to-one, and using the three-dimensional positions of the vertices of the refined mesh as the initial values ​​of the Gaussian point positions.

4. The digital human reconstruction method based on a mixture of grid and Gaussian point representation according to claim 1, characterized in that, Feature encoding of the Gaussian positions includes: A three-plane feature encoding network is used to encode the Gaussian positions, and high-resolution three-plane features are introduced into the facial region.

5. The digital human reconstruction method based on a mixture of grid and Gaussian point representation according to claim 1, characterized in that, Learning the geometric and color properties of Gaussian points includes: The feature encoding is performed by learning the geometric and appearance attributes of each Gaussian point through the geometric and appearance branches of the dual-branch neural network structure, respectively. The geometric and appearance branches are composed of two multilayer perceptrons (MLPs). The basic MLP is used to learn static identity features, and the pose-related MLP models non-rigid deformation by input pose parameters.

6. The digital human reconstruction method based on a mixture of grid and Gaussian point representation according to claim 1, characterized in that, The optimization constraints include: color consistency constraints, regularization constraints, and geometric consistency constraints; Wherein, the regularization constraint adopts Norm regularization is used to constrain the Gaussian position and scale offsets predicted by the geometric network; The geometric consistency constraints include consistency constraints based on normal maps and local and global consistency constraints based on depth maps.

7. A digital human reconstruction system based on a hybrid representation of grids and Gaussian points, used to implement the method as described in any one of claims 1-6, characterized in that, include: The mesh acquisition module is used to analyze and process human images in video datasets to obtain human mesh and pose parameters; The Gaussian binding module is used to subdivide the human body mesh, bind the vertices of the refined mesh to three-dimensional Gaussian points, and initialize the Gaussian positions in the normalized space. The feature encoding module is used to perform feature encoding on the Gaussian positions; The Gaussian attribute learning module is used to combine the feature encoding and pose parameters to learn the geometric and color attributes of Gaussian points and obtain a complete set of Gaussian attributes. The digital human construction module is used to generate digital human images based on the complete set of Gaussian attributes, using a linear blending skinning method combined with optimization constraints.

8. The digital human reconstruction system based on a hybrid representation of grids and Gaussian points according to claim 7, characterized in that, The Gaussian binding module includes: a mesh subdivision unit and a Gaussian binding unit; The mesh subdivision unit is used to subdivide the human body mesh using PyTorch3D; The Gaussian binding unit is used to bind the vertices of the refined mesh to three-dimensional Gaussian points in a one-to-one correspondence, and to use the three-dimensional position of the vertex of the refined mesh as the initial value of the position of the Gaussian point.

9. The digital human reconstruction system based on a hybrid representation of grids and Gaussian points according to claim 7, characterized in that, The Gaussian attribute learning module includes: a geometric network unit and a color network unit; The geometric network unit is used to learn the geometrically relevant attributes of each Gaussian point through the geometric network branches of the dual-branch neural network structure for the feature encoding; The color network unit is used to learn the appearance attributes of each Gaussian point through the appearance network branches of the dual-branch neural network structure.

10. The digital human reconstruction system based on a hybrid representation of grids and Gaussian points according to claim 7, characterized in that, The digital human building module includes: pose-driven unit, Gaussian rendering unit, color consistency unit, regularization constraint unit, and geometric consistency unit; The posture driving unit is used to transform the complete set of Gaussian attributes according to the posture parameters of the character in the current frame through a linear blending skinning method, thereby realizing posture driving. The Gaussian rendering unit is used to render the complete Gaussian properties using a three-dimensional Gaussian sputtering method to generate a high-quality digital human image. The color consistency unit is used to constrain the color consistency between the rendered image and the real image using RGB, SSIM, and LPIPS metrics. The regularization constraint unit is used to employ Norm regularization is used to constrain the Gaussian position and scale offsets predicted by the geometric network; The geometric consistency unit is used to implement consistency constraints based on normal maps and local and global consistency constraints based on depth maps.

Images