Monocular video dynamic human body reconstruction method and system based on three-dimensional gaussian splashing

Abstract

The application is suitable for the technical field of computer vision and computer graphics, and provides a monocular video dynamic human body reconstruction method and system based on three-dimensional Gaussian splashing, which comprises the following steps: data preprocessing; three-dimensional Gaussian initialization in canonical space; deforming the three-dimensional Gaussian in canonical space to an intermediate pose space to obtain a non-rigid deformation three-dimensional Gaussian and a pose-related feature; converting the non-rigid deformation three-dimensional Gaussian to an observation space by using linear mixed skinning to obtain an observation space three-dimensional Gaussian; decoding the color related to the perspective according to the Gaussian color feature, the pose-related feature and the perspective direction; constructing a total loss function comprising a normal consistency regularization term to optimize the three-dimensional Gaussian attribute and the network parameter; and rendering into a target human body image by a differentiable Gaussian splashing rasterizer.The application can realize fast and full-automatic reconstruction from monocular video to high-fidelity animatable human body model, and is suitable for virtual reality, film production and other fields.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and computer graphics technology, and in particular relates to a method and system for dynamic human body reconstruction in monocular video based on three-dimensional Gaussian splashing. Background Technology

[0002] Digital human body avatar technology is a key supporting technology for future applications in games, movies, augmented reality (AR), virtual reality (VR), and the "metaverse." High-fidelity 3D human body reconstruction can greatly enhance the realism and immersion of digital content. However, existing high-fidelity reconstruction solutions typically rely on expensive multi-camera acquisition systems, professional studio environments, or manual modeling by skilled artists, resulting in high costs and complex processes, making them difficult to popularize in ordinary consumer scenarios. Therefore, researching how to directly reconstruct high-fidelity dynamic human body models from monocular videos captured by portable devices (such as mobile phones) has significant practical application value.

[0003] In the field of 3D reconstruction technology, neural radiation field (NeRF)-based methods (such as NeuralBody and HumanNeRF) have achieved high-quality novel perspective image synthesis through implicit representations. However, these methods typically suffer from two inherent drawbacks: first, they are computationally expensive, requiring hours or even days of training time, and their inference speed is extremely slow, making real-time rendering difficult; second, they have poor editability, as implicit representations are difficult to directly edit and animate geometric or appearance attributes.

[0004] The recently proposed 3D Gaussian Splatting (3DGS) technique introduces a novel explicit scene representation method. It models scenes using a large number of learnable 3D Gaussian distributions and renders them using a splatting approach. This method achieves real-time rendering and rapid training while maintaining rendering quality comparable to or even better than NeRF. Based on this, subsequent research proposed 3DGS, applying it to dynamic human reconstruction. By decomposing human deformation into non-rigid and rigid parts, it avoids the computational overhead of NeRF while retaining the ability to fit high-quality image synthesis.

[0005] However, existing 3DGS solutions still face the following technical bottlenecks in practical applications: 1. Poor geometric quality: Due to the inconsistency of the 3D Gaussian splash depth, noisy surface normals will be generated during the rendering process, resulting in rough 3D geometric surfaces and loss of details, especially poor representation of fine structures such as clothing wrinkles.

[0006] 2. Reliance on manual preprocessing: This method relies heavily on multi-view images acquired by a multi-camera system or requires professional annotation of the dataset (such as foreground segmentation and SMPL parameter fitting), which exceeds the application scenarios of ordinary users and limits its feasibility of direct reconstruction from personal mobile phone videos.

[0007] 3. Weak monocular generalization ability: Under training conditions that only provide a single camera view, traditional color representation methods based on spherical harmonic functions are difficult to generalize to unseen test views.

[0008] Therefore, how to solve the above problems and realize a high-geometric-quality, fully automated, and efficient dynamic human body reconstruction method suitable for monocular input is a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0009] The purpose of this invention is to provide a method and system for dynamic human body reconstruction in monocular video based on three-dimensional Gaussian splashing, which aims to solve the technical problems of low geometric reconstruction quality, reliance on manual preprocessing, and weak monocular generalization ability in the prior art.

[0010] This invention is implemented as follows: a method for dynamic human body reconstruction in monocular video based on three-dimensional Gaussian splashing, comprising the following steps: Data preprocessing: Acquire monocular video containing human body and perform fully automatic preprocessing to obtain calibrated camera parameters, foreground mask and human body SMPL parameters; Normalized 3D Gaussian initialization: A set of 3D Gaussian distributions is initialized in normalized space based on the SMPL mesh to obtain the normalized 3D Gaussian distribution; Non-rigid deformation: By using the non-rigid deformation module and combining it with attitude coding, the standard space 3D Gaussian is deformed to the intermediate attitude space to obtain non-rigid deformed 3D Gaussian and attitude-related features. Rigid Deformation: Through the rigid deformation module, the non-rigid deformation 3D Gaussian is transformed into the observation space using linear hybrid skin, resulting in the observation space 3D Gaussian; Color processing: Based on a multilayer perceptron for color processing, the viewpoint-related color is decoded according to Gaussian color features, pose-related features, and viewpoint direction. Regularization constraints and optimization: Construct a total loss function that includes a normal consistency regularization term, optimize the 3D Gaussian properties and network parameters, and obtain the optimized 3D Gaussian of the observation space and its viewpoint-related color; Differentiable rendering: Using a differentiable Gaussian splash rasterizer, the optimized three-dimensional Gaussian of the observation space and its viewpoint-dependent colors are rendered as an image of the target human body.

[0011] Furthermore, the data preprocessing steps specifically include: Camera calibration: The structure-of-motion regression algorithm is used to solve for camera parameters in monocular video sequences; Image segmentation: Based on the SAM model, segmentation inference is performed on each frame of the monocular video to obtain the foreground mask; Human body parameterization: Based on the HybrIK-X model, full-body pose estimation is performed on each frame of the monocular video to obtain the human body SMPL parameters.

[0012] Furthermore, the specific method for camera calibration is as follows: feature point extraction and matching are performed on the monocular video sequence, and the camera intrinsic and extrinsic parameter matrices corresponding to each frame image are obtained through bundle adjustment iterative optimization.

[0013] Furthermore, the specific method for image segmentation is as follows: using a human detector to obtain bounding box cues of the human body region in each frame of the image; inputting the image and bounding box cues into the SAM model to output the corresponding human body region segmentation mask; performing morphological post-processing on the segmentation mask to obtain the foreground mask.

[0014] Furthermore, the specific method for human body parameterization is as follows: using a convolutional neural network based on the HybrIK-X model to extract 2D key point heatmaps of each frame of image; mapping 2D key points to 3D joint rotation angles using an inverse kinematics solver; and recovering complete human body SMPL parameters based on a preset SMPL parameterization model and the 3D joint rotation angles; the human body SMPL parameters include posture parameters and shape parameters.

[0015] Furthermore, the non-rigid deformation step specifically includes: The human SMPL parameters are encoded using a lightweight hierarchical posture encoder to obtain the posture latent code; The standard position and attitude latent code in the canonical space are input into the deformable network, which outputs Gaussian position, scaling, rotation offsets, and attitude-related features: The three-dimensional Gaussian properties after non-rigid deformation are calculated based on the Gaussian position, scaling, and rotation offset.

[0016] Furthermore, the rigid deformation step specifically includes: Using a neural skin network To predict location Skin weights at the location; given a set of rigid skeleton transformation matrices For the 24 joints of the SMPL, the forward transformation moments and observation space properties are calculated as follows: Rigid skeleton transformation matrix Used Each skeleton transformation is composed of a... The rotation-translation matrix representation is then passed through a set of skinning weights. Perform linear mixing to satisfy From coordinate-based neural skin field Modeling; therefore, the forward linear blending skin function is expressed as: ; Where, x c To standardize the location of spaces; Representing points in gauge space The corresponding point in the observation space; LBS(·) is the forward linear blending skin function; Then, the non-rigid deformation 3D Gaussian is transformed using the rigid deformation module. Transition to observation space: ; in, Represents a three-dimensional Gaussian in the observation space; This represents a set of rigid skeleton transformations accepted by a linear hybrid skinning function, and the multilayer perceptron learned for skinning. Used for prediction Positional skin weights; obtaining the 3D Gaussian rotation matrix through forward linear blending skin: ; in, It is the forward transformation matrix. It is composed of quaternions The resulting rotation matrix.

[0017] Furthermore, the color processing steps specifically include: Three-dimensional location is encoded as spatial features through a multi-layered hash grid; The spatial features are concatenated with Gaussian color features, pose-related features, each frame's latent encoding, and the normalized viewing direction to form a joint feature vector. The Gaussian color features are intrinsic color feature vectors learned from Gaussian points themselves, defined in the normalized space, and transferred to the observation space through deformation. The each frame's latent encoding assigns an independent latent encoding vector to each frame of the image to capture illumination changes and camera parameter differences in the monocular video sequence. The normalized viewing direction is a feature vector that maps the viewpoint direction vector to a high-dimensional space through a position encoding function. The joint feature vector is input into a color processing multilayer perceptron, which outputs a color that is related to the viewing angle.

[0018] Furthermore, the regularization constraint and optimization steps specifically include: In differentiable rendering, for a specific ray direction Intersecting 3D Gaussians, their surface normals It is approximated as the normal vector passing through the center of the Gaussian and perpendicular to the plane formed by the ray and the Gaussian scaling axis. The specific calculation formula is as follows: ; in, It is the rotation matrix of the Gaussian. It is its scaling vector. This indicates element-wise multiplication. This indicates taking the reciprocal of each element; Secondly, based on the estimation of reference surface normals using differentiable rendering, to provide a supervision signal for the calculated surface normals, the reference surface normals are estimated in real time through spatial gradient calculation using the intermediate depth map produced by differentiable rendering, and the depth map of the current viewpoint is obtained during rendering. Then, its position in the image space is calculated. and gradient in direction and Then, the true surface normal N of the object corresponding to that pixel position is estimated: ; ; in, and These are the Sobel kernels in the horizontal and vertical directions, respectively; Then, a normal consistency regularization loss function is constructed: the normal consistency constraint is introduced into the optimization objective of 3D Gaussian splashing, forcing the 3D Gaussian distribution to align and shrink from the interior towards the surface of the real object by minimizing the difference between the two types of normals; the normal consistency loss function... Defined as:

[0019] Where i is the index of the Gaussian intersection along the ray. This represents the mixed weights, where N is the estimated true normal to the object's surface. Define the loss function at equidistant positions respectively and equidistant covariance loss function for: ; ; in, This represents the number of three-dimensional Gaussians. Let k represent the k nearest neighbors of the i-th Gaussian point. It is a distance function; Let i and j represent the positions of the i-th and j-th Gaussian points in the normalized space, respectively. These represent the positions of the i-th and j-th Gaussian points in the observation space, respectively. Finally, construct the total loss function for multi-objective joint optimization. The details are as follows: ; in, It is a pixel-level L1 loss function; This is the perceptual loss function based on deep networks; L1 loss for foreground masking; To utilize the SMPL prior for regularization of skin weights; , , , , , , These are the weight coefficients of the corresponding loss functions.

[0020] Another objective of this invention is to provide a monocular video dynamic human body reconstruction system based on three-dimensional Gaussian splashing, for implementing the aforementioned monocular video dynamic human body reconstruction method based on three-dimensional Gaussian splashing, comprising: The data preprocessing module is used to acquire monocular videos containing human bodies and perform fully automatic preprocessing to obtain calibrated camera parameters, foreground mask, and human body SMPL parameters. The normalized space 3D Gaussian initialization module is used to initialize a set of 3D Gaussian distributions in normalized space based on the SMPL mesh, so as to obtain the normalized space 3D Gaussian. The non-rigid deformation module is used to combine attitude coding to deform the normed space 3D Gaussian to the intermediate attitude space, thereby obtaining non-rigid deformable 3D Gaussian and attitude-related features. The rigid deformation module is used to transform a non-rigid deformation 3D Gaussian into the observation space using linear hybrid skin, thus obtaining a 3D Gaussian in the observation space. The color processing module is used to decode the view-related color based on the color processing multilayer perceptron, according to Gaussian color features, pose-related features, and view direction. The regularization constraint and optimization module is used to construct a total loss function that includes a normal consistency regularization term, optimize the 3D Gaussian properties and network parameters, and obtain the optimized 3D Gaussian of the observation space and its viewpoint-related color. The differentiable rendering module is used to render the optimized three-dimensional Gaussian of the observation space and its viewpoint-dependent colors into a target human image using a differentiable Gaussian splash rasterizer.

[0021] The present invention provides a method for dynamic human body reconstruction based on monocular video using 3D Gaussian splashing. This method eliminates human intervention through fully automated preprocessing, suppresses surface noise caused by Gaussian depth inconsistency by employing normal consistency regularization constraints, and enhances the generalization ability under monocular viewpoint by combining color processing multilayer perceptron. This enables rapid and fully automated reconstruction from monocular video to a high-fidelity animable human body model. While significantly improving the quality of geometric reconstruction (especially details such as clothing folds), it also significantly improves training efficiency compared to the neural radiation field method and can generalize to unseen test viewpoints. It is applicable to fields such as virtual reality and film and television production. Attached Figure Description

[0022] Figure 1 This is a flowchart illustrating the method for monocular video dynamic human body reconstruction based on three-dimensional Gaussian splashing provided in an embodiment of the present invention.

[0023] Figure 2 A visual comparison between the training viewpoint image A and the rendered result B under the new view.

[0024] Figure 3 This is a diagram showing the results of human image segmentation based on the SAM model.

[0025] Figure 4 This is a graph showing the parameterized results of the human body based on the HybrIK-X model.

[0026] Figure 5 The rendering results are the real image A and the new perspective image B. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0028] like Figure 1 As shown, in one embodiment of the present invention, a method for dynamic human body reconstruction based on three-dimensional Gaussian splashing in monocular video is provided, which can be used to quickly reconstruct a high-fidelity, animable digital human body from monocular video, specifically including the following steps: S1. Data preprocessing: Acquire monocular video containing human body and perform fully automatic preprocessing to obtain calibrated camera parameters, foreground mask and human body SMPL parameters. S2. Initialization of three-dimensional Gaussian distribution in normal space: Based on the SMPL mesh, a set of three-dimensional Gaussian distributions is initialized in normal space to obtain the three-dimensional Gaussian distribution in normal space; S3. Non-rigid deformation: Through the non-rigid deformation module, combined with attitude coding, the standard space three-dimensional Gaussian is deformed to the intermediate attitude space to obtain non-rigid deformed three-dimensional Gaussian and attitude-related features. S4. Rigid Deformation: Through the rigid deformation module, the non-rigid deformation 3D Gaussian is transformed into the observation space using linear hybrid skin to obtain the observation space 3D Gaussian. S5. Color Processing: Based on a multilayer perceptron for color processing, the color related to the viewing angle is decoded according to Gaussian color features, pose-related features, and viewing direction. S6. Regularization Constraints and Optimization: Construct a total loss function that includes a normal consistency regularization term, optimize the 3D Gaussian properties and network parameters, and obtain the optimized 3D Gaussian of the observation space and its viewpoint-related color. S7. Differentiable rendering: Using a differentiable Gaussian splash rasterizer, the optimized three-dimensional Gaussian of the observation space and its viewpoint-dependent colors are rendered as a target human image.

[0029] In a preferred embodiment of the present invention, the input to step S1 is unprocessed monocular RGB video. This step is performed through an end-to-end fully automated preprocessing pipeline, requiring no manual intervention, no multi-view acquisition equipment, and no professional calibration tools, thus achieving automated conversion from raw video to training data. This fully automated preprocessing pipeline consists of three cooperating modules: a camera calibration module, an image segmentation module, and a human body parameterization module. Step S1 specifically includes: S11. Camera Calibration: The structure-of-motion regression algorithm is used to solve for camera parameters in monocular video sequences. S12. Image segmentation: Based on the SAM model, segmentation inference is performed on each frame of the monocular video to obtain the foreground mask. S13. Human body parameterization: Based on the HybrIK-X model, full-body pose estimation is performed on each frame of the monocular video to obtain the human body SMPL parameters.

[0030] Specifically, the camera calibration method is as follows: The Structure from Motion (SfM) algorithm is used to solve for camera parameters in the monocular video sequence; specifically, feature point extraction and matching are performed on the monocular video sequence, and iterative optimization is achieved through bundle adjustment to obtain the camera intrinsic parameter matrix corresponding to each frame. (Including focal length) and the main point ) and extrinsic parameter matrix (Including rotation matrix) Translation vector This method eliminates the need for a pre-set calibration board or a known scene structure, enabling fully automatic camera calibration under monocular video conditions.

[0031] The specific method for image segmentation is as follows: The Segment Anything Model (SAM) is used to infer the meaning of each frame of the image. The SAM model is a segmentation foundation model pre-trained on a large-scale image dataset, possessing powerful zero-shot generalization capabilities. This embodiment of the invention can generate a high-quality foreground mask through the following steps: First, a human detector is used to obtain bounding box cues of the human region in each frame of the image; then, the image and bounding box cues are input into the SAM model, which outputs the corresponding human region segmentation mask; finally, morphological post-processing (such as closing operations to fill holes and opening operations to remove noise) is performed on the segmentation mask to obtain an accurate foreground mask. This method requires no manual annotation and can be adapted to various scenarios, poses, and clothing types.

[0032] The specific method for human body parameterization is as follows: The HybrIK-X model is used to perform full-body pose estimation for each frame of the image. The HybrIK-X model is a deep learning-based monocular human body parameter estimation model. Its core innovation lies in accurately converting 2D keypoint detection results into 3D joint rotations through inverse kinematics. The specific process is as follows: First, a convolutional neural network is used to extract 2D keypoint heatmaps from the image; then, an inverse kinematics solver maps the 2D keypoints to 3D joint rotation angles; finally, based on a pre-defined SMPL parameterization model, the complete human body SMPL parameters are recovered from the 3D joint rotation angles. The human body SMPL parameters include pose parameters. (3-axis rotation of 24 joints) and shape parameters (A 10-dimensional vector describing human height, weight, and build). This method can accurately estimate human body parameters from a monocular perspective, without the need for multi-view systems or manual annotation.

[0033] The three modules work together through the following mechanisms: the camera parameters output by the camera calibration module are used for coordinate system transformation in subsequent rendering; the foreground mask output by the image segmentation module is used for mask loss calculation, constraining the rendering results to be valid only within the human body region; and the human body parameterization module outputs human body SMPL parameters for normalized space Gaussian initialization and subsequent deformation driving. The outputs of the three modules together constitute a complete training dataset.

[0034] This step generates a complete dataset that can be directly used for training, including calibrated camera parameters, fitted human SMPL parameters, and foreground masks.

[0035] In a preferred embodiment of the present invention, step S2 specifically includes: randomly sampling on the SMPL grid surface in the canonical space (typically T-pose). A point, used as a standard three-dimensional Gaussian point. Initialization, where each Gaussian point has a position Scaling Rotation (represented by quaternions), opacity Gaussian color characteristics Learnable attributes, etc.

[0036] In a preferred embodiment of the present invention, step S3 specifically includes: S31. Use a lightweight hierarchical posture encoder to encode the human body SMPL parameters to obtain the posture latent code; S32. Input the standard position and attitude latent code in the canonical space into the deformable network, and output the Gaussian position, scaling, rotation offsets, and attitude-related features: S33. Calculate the three-dimensional Gaussian properties after non-rigid deformation based on the Gaussian position, scaling, and rotation offset.

[0037] In practical applications, to model non-rigid clothing deformation related to posture, this embodiment of the invention introduces a non-rigid deformation module, wherein the expression for non-rigid deformation is as follows: ; In the formula, Represents a three-dimensional Gaussian in gauge space. Represents a non-rigid deformation three-dimensional Gaussian. Learnable parameters representing non-rigid deformation modules. For attitude latent coding, it uses a lightweight hierarchical attitude encoder for SMPL pose and shape. Encoding is performed. Specifically, deformable networks. Standard position and attitude code As input, output the Gaussian position, scale, and rotation offsets. and eigenvectors : ; This invention uses a multi-layer hash grid to encode 3D position into spatial features, and then combines them with the constructed attitude latent code. Connect and input a shallow multilayer perceptron with two hidden layers and a width of 128. A typical Gaussian transformation is: ; in, Represents the position, scaling, and rotation of a standard 3D Gaussian; Represents the position, scaling, and rotation of a non-rigidly deformable 3D Gaussian; for The quaternion representation of ; the ⋅ operator on a quaternion is equivalent to multiplying two rotation matrices derived from two quaternions. Because quaternions Corresponding to the unit rotation matrix, therefore when At that time, there is .

[0038] In a preferred embodiment of the present invention, step S4 specifically includes: This step employs Linear Hybrid Skin (LBS) technology to transform non-rigid deformable 3D Gaussian... Transform to the observation space to obtain the three-dimensional Gaussian of the observation space. Specifically, firstly, a neural skin network is used. To predict location Skin weights at the location; given a set of rigid skeleton transformation matrices For the 24 joints of the SMPL, the forward transformation moments and observation space properties are calculated as follows: Rigid skeleton transformation matrix Used Each skeleton transformation is composed of a... The rotation-translation matrix representation is then passed through a set of skinning weights. Perform linear mixing to satisfy From coordinate-based neural skin field Modeling; therefore, the forward linear blending skin function is expressed as: ; Where, x c To standardize the location of spaces; Representing points in gauge space The corresponding point in the observation space; LBS(·) is the forward linear blending skin function; Then, the non-rigid deformation 3D Gaussian is transformed using the rigid deformation module. Transition to observation space: ; in, Represents a three-dimensional Gaussian in the observation space; This represents a set of rigid skeleton transformations accepted by a linear hybrid skinning function, and the multilayer perceptron learned for skinning. Used for prediction Skin weights at position; in this embodiment of the invention, a three-dimensional Gaussian rotation matrix is ​​obtained through forward linear blending skinning: ; in, It is the forward transformation matrix. It is composed of quaternions The resulting rotation matrix.

[0039] In a preferred embodiment of the present invention, step S5 specifically includes: To address the poor generalization ability of spherical harmonic functions under monocular conditions, this invention employs a multi-layer perceptron (MLP) for color processing to decode colors. This method, by introducing pose-related features and illumination encoding, achieves generalized rendering for unseen test views under monocular training conditions. Specifically, it first uses a multi-resolution hash grid to represent the 3D positions... Encoding as spatial features Next, spatial features Gaussian color characteristics Posture-related features Latent coding per frame and standardized observation directions The joint feature vector is then connected to form a joint feature vector; subsequently, the joint feature vector is input into a color processing multilayer perceptron, which outputs a view-dependent color c.

[0040] in, Multilayer perceptron for color processing; Gaussian color characteristics It is an intrinsic color feature vector learned from the Gaussian point itself, defined in the canonical space and transmitted to the observation space through deformation, serving as the basis for color decoding; pose-related features Outputted by the non-rigid deformation module, it encodes local geometric deformation information under the current pose, enabling color to adaptively adjust with pose changes (e.g., shadow changes in clothing folds); each frame's latent encoding An independent latent encoding vector is assigned to each frame of the image to capture illumination changes and camera parameter differences in the monocular video sequence, enabling the model to adapt to illumination fluctuations between different frames during the joint optimization process. To determine the three-dimensional position through a multi-layered hash grid Spatial features obtained through encoding; To use the position encoding function View direction vector Feature vectors mapped to a high-dimensional space. The five types of features mentioned above are concatenated and fused, then input into a small MLP with two hidden layers and a width of 128, ultimately decoding to obtain the viewpoint-dependent RGB color. .

[0041] Before being input into the MLP, the five types of features mentioned above are fused through a concatenation operation. Specifically, the three-dimensional positions are first fused using a multi-layer hash grid. Encoding as spatial features Then Gaussian color characteristics Posture-related features Latent coding per frame and the normalized observation direction after location encoding The features are concatenated to form a joint feature vector. This joint feature vector is input into a small MLP with two hidden layers and a width of 128, and the final output is a view-dependent color. (RGB color values).

[0042] In a preferred embodiment of the present invention, step S6 specifically includes: To improve geometric quality, this invention introduces normal consistency regularization for 3D Gaussian representation, which is the core improvement of this invention to address the low geometric quality of traditional 3DGS methods.

[0043] Specifically, addressing the lack of a clear definition of surface normals in three-dimensional Gaussian distributions, this invention proposes an approximate normal calculation method based on ray-Gaussian intersection geometry. Firstly, during differentiable rendering, for a specific ray direction... Intersecting 3D Gaussians, their surface normals It is approximated as the normal vector passing through the center of the Gaussian and perpendicular to the plane formed by the ray and the Gaussian scaling axis. The specific calculation formula is as follows: ; in, It is the rotation matrix of the Gaussian. It is its scaling vector. This indicates element-wise multiplication. This indicates taking the reciprocal of each element; the formula obtains a local normal estimate aligned with Gaussian geometric properties by transforming the ray direction from world space to a Gaussian local scaling space and then mapping it back to world space. Secondly, based on the estimation of reference surface normals using differentiable rendering, to provide a supervision signal for the calculated surface normals, the reference surface normals are estimated in real time through spatial gradient calculations using the intermediate product of differentiable rendering—the depth map—and the depth map of the current viewpoint is obtained during rendering. Then, its position in the image space is calculated. and gradient in direction and Then, the true surface normal N of the object corresponding to that pixel position is estimated: ; ; in, and These are Sobel kernels in the horizontal and vertical directions, respectively. This method requires no additional networks or annotations and can be obtained solely through the intrinsic computation graph of differentiable rendering, achieving efficient "self-supervised" geometric constraints. Then, a normal consistency regularization loss function is constructed: the normal consistency constraint is introduced into the optimization objective of 3D Gaussian splashing, forcing the 3D Gaussian distribution to align and shrink from the interior towards the surface of the real object by minimizing the difference between the two types of normals; the normal consistency loss function... Defined as:

[0044] Where i is the index of the Gaussian intersection along the ray. The weighting is denoted by , and N is the estimated true normal of the object surface. This loss function effectively suppresses geometric surface noise caused by the inconsistency of the depth of 3D Gaussian splash by forcing the Gaussian point normal to be consistent with the normal direction of the rendered surface. At the same time, the normal consistency loss function only constrains the consistency of the normal direction and does not directly penalize position changes. Therefore, it can completely preserve high-frequency geometric details such as clothing wrinkles while suppressing noise. Furthermore, an equidistant constraint is introduced to enhance generalization ability under sparse input. Addressing the challenge of observing only a single viewpoint and struggling to constrain non-rigid deformation under monocular video input, this invention introduces an equidistant constraint. When using monocular video as input, only a portion of the human body's viewpoint is visible in each frame, leading to insufficient constraints on the non-rigid deformation network, and noise easily generated during deformation from the normal space to the observation space. To solve this problem, this invention utilizes an equidistant constraint to limit the similarity of adjacent 3D Gaussian centers after deformation, and extends this constraint to the covariance matrix of the Gaussian; specifically, the equidistant constraint includes an equidistant position loss function. and equidistant covariance loss function Define the loss function for equidistant positions respectively. and equidistant covariance loss function for: ; ; in, This represents the number of three-dimensional Gaussians. Let k represent the k nearest neighbors of the i-th Gaussian point (k is 5 in this embodiment). The distance function (using the L2 norm); Let i and j represent the positions of the i-th and j-th Gaussian points in the normalized space, respectively. These represent the positions of the i-th and j-th Gaussian points in the observation space, respectively. , Let represent the covariance matrices of the i-th and j-th Gaussian points in the normal space, respectively; , Let i and j represent the covariance matrices of the i-th and j-th Gaussian points in the observation space, respectively; subscripts are used. and These represent the properties of the norm space and the observation space, respectively. The equidistant constraint effectively suppresses deformation noise caused by input sparsity by constraining the relative distance and shape similarity of adjacent Gaussian points before and after deformation, thereby enhancing the model's generalization ability to unseen viewpoints and new poses.

[0045] Finally, construct the total loss function for multi-objective joint optimization. This ensures that while improving geometric quality, the realism of texture rendering is not sacrificed, as detailed below: ; in, It is a pixel-level L1 loss function; This is the perceptual loss function based on deep networks; L1 loss for foreground masking; To utilize the SMPL prior for regularization of skin weights; , , , , , , These are the weight coefficients of the corresponding loss functions.

[0046] In a preferred embodiment of the present invention, step S7 specifically includes: The optimized observation space Gaussian The color c, which is related to the viewpoint, is input into a differentiable Gaussian splash rasterizer. The Gaussian is projected onto the 2D image plane through a fast rasterization algorithm and blended in depth order to finally render the target human image.

[0047] In another embodiment of the present invention, a monocular video dynamic human body reconstruction system based on three-dimensional Gaussian splashing is also provided to implement the above-mentioned monocular video dynamic human body reconstruction method based on three-dimensional Gaussian splashing, comprising: The data preprocessing module is used to acquire monocular videos containing human bodies and perform fully automatic preprocessing to obtain calibrated camera parameters, foreground mask, and human body SMPL parameters. The normalized space 3D Gaussian initialization module is used to initialize a set of 3D Gaussian distributions in normalized space based on the SMPL mesh, so as to obtain the normalized space 3D Gaussian. The non-rigid deformation module is used to combine attitude coding to deform the normed space 3D Gaussian to the intermediate attitude space, thereby obtaining non-rigid deformable 3D Gaussian and attitude-related features. The rigid deformation module is used to transform a non-rigid deformation 3D Gaussian into the observation space using linear hybrid skin, thus obtaining a 3D Gaussian in the observation space. The color processing module is used to decode the view-related color based on the color processing multilayer perceptron, according to Gaussian color features, pose-related features, and view direction. The regularization constraint and optimization module is used to construct a total loss function that includes a normal consistency regularization term, optimize the 3D Gaussian properties and network parameters, and obtain the optimized 3D Gaussian of the observation space and its viewpoint-related color. The differentiable rendering module is used to render the optimized three-dimensional Gaussian of the observation space and its viewpoint-dependent colors into a target human image using a differentiable Gaussian splash rasterizer.

[0048] Example: To verify the effectiveness and superiority of the method proposed in this embodiment, this example selects several mainstream dynamic human reconstruction methods and recently published reconstruction methods based on neural radiation fields and three-dimensional Gaussian splashing as comparison benchmarks, and performs quantitative and qualitative analysis on the public dataset ZJU-MoCap. Simultaneously, experiments on the custom dataset JLU-JingMing verify the robustness and generalization ability of the fully automated preprocessing workflow provided in this embodiment.

[0049] Dataset: This embodiment uses the LightStage subset from the ZJU-MoCap dataset as the primary evaluation benchmark. This dataset contains multiple dynamic human videos captured using a multi-camera system, providing SMPL-X parameters recovered using EasyMocap, covering complex movements of the body, hands, and face, such as rotation, Tai Chi, punching, and kicking, providing rich pose variations and scene diversity for dynamic human reconstruction.

[0050] Comparison Methods: The following representative methods were selected as the comparison benchmarks in this embodiment: 1. Methods based on neural radiation fields: NeuralBody, HumanNeRF, MonoHuman, ARAH; 2. A method based on three-dimensional Gaussian splashing: 3DGS; Evaluation metrics: This embodiment uses the following objective evaluation metrics to quantitatively assess the rendering quality of the reconstruction results: Peak Signal-to-Noise Ratio (PSNR): Measures the pixel-level difference between the reconstructed image and the real image; a higher value indicates better image quality. Structural similarity (SSIM): Measures the similarity of images in terms of brightness, contrast, and structure. The closer the value is to 1, the better the image quality. Perceptual Similarity (LPIPS): Calculates the perceptual difference between images based on deep network features. The lower the value, the closer the image is to the real image in visual perception. To objectively evaluate the performance of the method of this invention, a quantitative comparative experiment was conducted on the ZJU-MoCap dataset, and the results are shown in Table 1.

[0051] Table 1. Quantitative comparison results of different methods on the ZJU-MoCap dataset.

[0052] The quantitative results shown in Table 1 can be seen as follows: Significantly improved training efficiency: The method of this invention requires only 0.5 hours of GPU training time, which is two orders of magnitude higher than the NeRF-based method (NeuralBody requires 12 hours, HumanNeRF, ARAH, etc. require several days). This is due to the explicit representation and differentiable rendering mechanism of 3D Gaussian splashing, which avoids the time-consuming ray propagation calculation in the NeRF method. Outstanding real-time inference capability: The method of this invention achieves a real-time rendering speed of 50 FPS, far exceeding all NeRF-based comparison methods (0.12-2 FPS), meeting the stringent low-latency requirements of applications such as virtual reality and real-time interaction; Superior Rendering Quality Across the Board: In terms of PSNR, the method of this invention achieves 30.86dB, superior to 3DGS (30.64dB) and HumanNeRF (30.41dB), and slightly higher than ARAH (30.85dB), indicating an advantage in pixel-level reconstruction accuracy. In terms of SSIM, the method achieves 0.9777, comparable to the best method. Particularly noteworthy is the perceptual similarity LPIPS, where the method of this invention significantly outperforms all comparison methods with a score of 20.59, reducing the similarity by 1.4% compared to 3DGS and 14.4% compared to HumanNeRF, fully validating the significant effect of the normal consistency regularization introduced in this invention on improving visual perception quality. Qualitative Analysis: To further demonstrate the visual effects of the method of this invention, this embodiment qualitatively compares the new view synthesis results of different methods on the ZJU-MoCap dataset, such as... Figure 2 As shown. From Figure 2 The qualitative comparison shown can be seen as follows: 1. Fidelity of Clothing Details: When dealing with complex clothing folds, the method of this invention can more accurately capture the natural deformation and subtle texture of the fabric. For example, when a person is performing a "Tai Chi" movement, the dynamic folds created by loose clothing are completely preserved by the method of this invention, while the 3DGS method shows a certain degree of blurring and geometric noise, and the NeRF-based method, although the overall outline is correct, lacks sufficient detail sharpness.

[0053] 2. Geometric Surface Smoothness: The surface of the human figure generated by the method of this invention is smoother and more natural, especially at the edges of the limbs and the junction of the torso, where artifacts and noise are significantly reduced. This directly verifies the effectiveness of normal consistency regularization in suppressing Gaussian depth inconsistency and forcing 3D Gaussians to align with the real surface.

[0054] 3. Adaptability to complex postures: In scenarios with rapid and large-amplitude movements such as "punching" and "kicking", the method of this invention can still maintain stable reconstruction quality without obvious ghosting or deformation, demonstrating its strong adaptability to dynamic scenes.

[0055] To verify the effectiveness of the fully automated preprocessing workflow of this invention in practical application scenarios, experiments were conducted on the custom JLU-JingMing dataset. This dataset was captured by a regular mobile phone in a complex indoor environment, with background elements including doors, windows, and floors, naturally varying lighting conditions, and the subject rotating 360°.

[0056] Preprocessing results: such as Figure 3 and Figure 4 As shown, the SAM model integrated in this embodiment of the invention successfully segmented a high-precision human foreground mask from a complex background; simultaneously, the HybrIK-X model accurately estimated the full-body pose of the person, and the recovered SMPL mesh highly matched the pose of the person in the original image. The entire preprocessing process is fully automated and requires no manual intervention.

[0057] Reconstruction Results: After fully automated preprocessing, the generated data is input into the method of this invention for training. Experimental results show that even under non-ideal conditions (complex background, natural lighting, monocular input), the method of this invention can still accurately reconstruct the 3D pose and appearance details of a person, such as... Figure 5 As shown, the new perspective synthesis results retain the natural folds and textures of clothing, and the character outlines are clear with no obvious artifacts, fully verifying the robustness and generalization ability of the invention in practical application scenarios.

[0058] In summary, through systematic quantitative comparison and qualitative analysis, the method proposed in this embodiment of the invention demonstrates significant advantages in training efficiency, inference speed, rendering quality, and geometric accuracy, fully verifying its effectiveness and superiority. Specifically, it includes the following beneficial effects: 1. High-quality geometric reconstruction: By introducing normal consistency regularization for 3D Gaussian properties, the surface noise caused by inconsistent Gaussian splash depth is effectively suppressed, significantly improving the quality of geometric reconstruction and making local details such as clothing wrinkles more realistic.

[0059] 2. Fully automated, low-barrier data processing: Integrating SAM and HybrIK-X models, an end-to-end fully automated preprocessing workflow is constructed. Users only need to provide monocular video taken with a mobile phone or camera, and the system can automatically complete camera calibration, human segmentation, and SMPL parameter estimation without manual intervention, greatly reducing the technical barrier to entry.

[0060] 3. High efficiency and strong generalization: It inherits the advantages of real-time rendering and fast training of 3DGS, with a training speed two orders of magnitude faster than NeRF-like methods. At the same time, it adopts MLP for color processing instead of traditional spherical harmonic functions, which improves the generalized rendering quality for unseen viewpoints under monocular training conditions.

[0061] 4. Complete system with high practicality: It provides a complete solution from raw video input to animated 3D model output, and can save intermediate models. It has good repeatability and versatility, and can be widely used in virtual try-on, remote interaction, film and television production and other fields.

[0062] It should be noted that each of the above modules can be implemented as a computer program, which can run on a computer device. The computer device's memory can store the computer program that makes up each module, enabling the processor to execute each step of the above method.

[0063] It should be understood that although the steps in the flowcharts of the various embodiments of the present invention are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the various embodiments may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least a portion of the sub-steps or stages of other steps.

[0064] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory.

[0065] The above embodiments merely illustrate several implementation methods of the present invention, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of this patent should be determined by the appended claims.

Claims

1. A method for dynamic human body reconstruction from monocular video based on three-dimensional Gaussian splashing, characterized in that, Includes the following steps: Data preprocessing: Acquire monocular video containing human body and perform fully automatic preprocessing to obtain calibrated camera parameters, foreground mask and human body SMPL parameters; Normalized 3D Gaussian initialization: A set of 3D Gaussian distributions is initialized in normalized space based on the SMPL mesh to obtain the normalized 3D Gaussian distribution; Non-rigid deformation: By using the non-rigid deformation module and combining it with attitude coding, the standard space 3D Gaussian is deformed to the intermediate attitude space to obtain non-rigid deformed 3D Gaussian and attitude-related features. Rigid Deformation: Through the rigid deformation module, the non-rigid deformation 3D Gaussian is transformed into the observation space using linear hybrid skin, resulting in the observation space 3D Gaussian; Color processing: Based on a multilayer perceptron for color processing, the viewpoint-related color is decoded according to Gaussian color features, pose-related features, and viewpoint direction. Regularization constraints and optimization: Construct a total loss function that includes a normal consistency regularization term, optimize the 3D Gaussian properties and network parameters, and obtain the optimized 3D Gaussian of the observation space and its viewpoint-related color; Differentiable rendering: Using a differentiable Gaussian splash rasterizer, the optimized three-dimensional Gaussian of the observation space and its viewpoint-dependent colors are rendered as an image of the target human body.

2. The method for monocular video dynamic human body reconstruction based on three-dimensional Gaussian splashing according to claim 1, characterized in that, The data preprocessing steps specifically include: Camera calibration: The structure-of-motion regression algorithm is used to solve for camera parameters in monocular video sequences; Image segmentation: Based on the SAM model, segmentation inference is performed on each frame of the monocular video to obtain the foreground mask; Human body parameterization: Based on the HybrIK-X model, full-body pose estimation is performed on each frame of the monocular video to obtain the human body SMPL parameters.

3. The method for monocular video dynamic human body reconstruction based on three-dimensional Gaussian splashing according to claim 2, characterized in that, The specific method for camera calibration is as follows: feature point extraction and matching are performed on the monocular video sequence, and the camera intrinsic and extrinsic parameter matrices are obtained for each frame image through bundle adjustment iterative optimization.

4. The method for monocular video dynamic human body reconstruction based on three-dimensional Gaussian splashing according to claim 2, characterized in that, The specific method for image segmentation is as follows: using a human detector to obtain bounding box cues of the human body region in each frame of the image; inputting the image and bounding box cues into the SAM model to output the corresponding human body region segmentation mask; performing morphological post-processing on the segmentation mask to obtain the foreground mask.

5. The method for monocular video dynamic human body reconstruction based on three-dimensional Gaussian splashing according to claim 2, characterized in that, The specific method for human body parameterization is as follows: using a convolutional neural network based on the HybrIK-X model to extract 2D key point heatmaps of each frame of image; mapping 2D key points to 3D joint rotation angles using an inverse kinematics solver; and recovering complete human body SMPL parameters based on a preset SMPL parameterization model and the 3D joint rotation angles; the human body SMPL parameters include posture parameters and shape parameters.

6. The method for monocular video dynamic human body reconstruction based on three-dimensional Gaussian splashing according to claim 1, characterized in that, The non-rigid deformation steps specifically include: The human SMPL parameters are encoded using a lightweight hierarchical posture encoder to obtain the posture latent code; The standard position and attitude latent code in the canonical space are input into the deformable network, which outputs Gaussian position, scaling, rotation offsets, and attitude-related features: The three-dimensional Gaussian properties after non-rigid deformation are calculated based on the Gaussian position, scaling, and rotation offset.

7. The method for monocular video dynamic human body reconstruction based on three-dimensional Gaussian splashing according to claim 1, characterized in that, The rigid deformation step specifically includes: Using a neural skin network To predict location Skin weights at the location; given a set of rigid skeleton transformation matrices For the 24 joints of the SMPL, the forward transformation moments and observation space properties are calculated as follows: Rigid skeleton transformation matrix Used Each skeleton transformation is composed of a... The rotation-translation matrix representation is then passed through a set of skinning weights. Perform linear mixing to satisfy From coordinate-based neural skin field Modeling; therefore, the forward linear blending skin function is expressed as: ； Where, x c To standardize the location of spaces; Representing points in gauge space The corresponding point in the observation space; LBS(·) is the forward linear blending skin function; Then, the non-rigid deformation 3D Gaussian is transformed using the rigid deformation module. Transition to observation space: ； in, Represents a three-dimensional Gaussian in the observation space; This represents a set of rigid skeleton transformations accepted by a linear hybrid skinning function, and the multilayer perceptron learned for skinning. Used for prediction Positional skin weights; obtaining the 3D Gaussian rotation matrix through forward linear blending skin: ； in, It is the forward transformation matrix. It is composed of quaternions The resulting rotation matrix.

8. The method for monocular video dynamic human body reconstruction based on three-dimensional Gaussian splashing according to claim 1, characterized in that, The color processing steps specifically include: Three-dimensional location is encoded as spatial features through a multi-layered hash grid; The spatial features are concatenated with Gaussian color features, pose-related features, each frame's latent encoding, and the normalized viewing direction to form a joint feature vector. The Gaussian color features are intrinsic color feature vectors learned from Gaussian points themselves, defined in the normalized space, and transferred to the observation space through deformation. The each frame's latent encoding assigns an independent latent encoding vector to each frame of the image to capture illumination changes and camera parameter differences in the monocular video sequence. The normalized viewing direction is a feature vector that maps the viewpoint direction vector to a high-dimensional space through a position encoding function. The joint feature vector is input into a color processing multilayer perceptron, which outputs a color that is related to the viewing angle.

9. The method for monocular video dynamic human body reconstruction based on three-dimensional Gaussian splashing according to claim 1, characterized in that, The regularization constraint and optimization steps specifically include: In differentiable rendering, for a specific ray direction Intersecting 3D Gaussians, their surface normals It is approximated as the normal vector passing through the center of the Gaussian and perpendicular to the plane formed by the ray and the Gaussian scaling axis. The specific calculation formula is as follows: ； in, It is the rotation matrix of the Gaussian. It is its scaling vector. This indicates element-wise multiplication. This indicates taking the reciprocal of each element; Secondly, based on the estimation of reference surface normals using differentiable rendering, to provide a supervision signal for the calculated surface normals, the reference surface normals are estimated in real time through spatial gradient calculation using the intermediate depth map produced by differentiable rendering, and the depth map of the current viewpoint is obtained during rendering. Then, its position in the image space is calculated. and gradient in direction and Then, the true surface normal N of the object corresponding to that pixel position is estimated: ； ； in, and These are the Sobel kernels in the horizontal and vertical directions, respectively; Then, a normal consistency regularization loss function is constructed: the normal consistency constraint is introduced into the optimization objective of 3D Gaussian splashing, forcing the 3D Gaussian distribution to align and shrink from the interior towards the surface of the real object by minimizing the difference between the two types of normals; the normal consistency loss function... Defined as: ， Where i is the index of the Gaussian intersection along the ray. This represents the mixed weights, where N is the estimated true normal to the object's surface. Define the loss function at equidistant positions respectively and equidistant covariance loss function for: ； ； in, This represents the number of three-dimensional Gaussians. Let k represent the k nearest neighbors of the i-th Gaussian point. It is a distance function; Let i and j represent the positions of the i-th and j-th Gaussian points in the normalized space, respectively. These represent the positions of the i-th and j-th Gaussian points in the observation space, respectively. , Let represent the covariance matrices of the i-th and j-th Gaussian points in the normal space, respectively; , Let represent the covariance matrices of the i-th and j-th Gaussian points in the observation space, respectively; Finally, construct the total loss function for multi-objective joint optimization. The details are as follows: ； in, It is a pixel-level L1 loss function; This is the perceptual loss function based on deep networks; L1 loss for foreground masking; To utilize the SMPL prior for regularization of skin weights; , , , , , , These are the weight coefficients of the corresponding loss functions.

10. A monocular video dynamic human body reconstruction system based on three-dimensional Gaussian splashing, used to implement the monocular video dynamic human body reconstruction method based on three-dimensional Gaussian splashing as described in any one of claims 1-9, characterized in that, include: The data preprocessing module is used to acquire monocular videos containing human bodies and perform fully automatic preprocessing to obtain calibrated camera parameters, foreground mask, and human body SMPL parameters. The normalized space 3D Gaussian initialization module is used to initialize a set of 3D Gaussian distributions in normalized space based on the SMPL mesh, so as to obtain the normalized space 3D Gaussian. The non-rigid deformation module is used to combine attitude coding to deform the normed space 3D Gaussian to the intermediate attitude space, thereby obtaining non-rigid deformable 3D Gaussian and attitude-related features. The rigid deformation module is used to transform a non-rigid deformation 3D Gaussian into the observation space using linear hybrid skin, thus obtaining a 3D Gaussian in the observation space. The color processing module is used to decode the view-related color based on the color processing multilayer perceptron, according to Gaussian color features, pose-related features, and view direction. The regularization constraint and optimization module is used to construct a total loss function that includes a normal consistency regularization term, optimize the 3D Gaussian properties and network parameters, and obtain the optimized 3D Gaussian of the observation space and its viewpoint-related color. The differentiable rendering module is used to render the optimized three-dimensional Gaussian of the observation space and its viewpoint-dependent colors into a target human image using a differentiable Gaussian splash rasterizer.

Images