Human body surface dynamic reconstruction method based on monocular video

Abstract

The invention discloses a human body surface dynamic reconstruction method based on a monocular video, and the method comprises the steps: extracting a key frame of human body motion from the monocular video, and obtaining an RGB image of the key frame, a mask, parameters of an SMPL human body template, and internal and external parameters of a camera; extracting a real normal vector diagram corresponding to the key frame, and constructing a basic data set; the basic data set is used for training a multi-stage progressive human body reconstruction framework, the framework comprises human body overall motion modeling serving as a first stage, local surface dynamic deformation modeling serving as a second stage and surface appearance and illumination modeling serving as a third stage, and an optimal reconstruction model is obtained through training; and extracting a new view angle frame of human body motion from the monocular video, obtaining an RGB image of the new view angle frame, parameters of the SMPL human body template and internal and external parameters of the camera, and inputting the RGB image, the parameters of the SMPL human body template and the internal and external parameters of the camera into the optimal reconstruction model to generate a human body dynamic reconstruction result under the conditions of a new view angle, a new posture and new illumination, thereby realizing high-quality rendering output with geometric details and appearance consistency.

Description

Technical Field

[0001] This invention relates to the technical field of three-dimensional human body reconstruction in computer vision, and in particular to a method for dynamic reconstruction of human body surface based on monocular video. Background Technology

[0002] Reconstructing dynamic 3D human models from monocular video is a core supporting technology for applications such as virtual digital humans, augmented reality, and film and television special effects. However, monocular input only provides two-dimensional visual cues, and traditional methods struggle to accurately reproduce fine-grained surface dynamics driven by posture, such as clothing wrinkles and muscle undulations. These details directly determine the realism of the model and are key prerequisites for new perspective rendering, new pose generalization, and new lighting re-rendering.

[0003] Existing methods are mainly divided into implicit representation methods based on neural radiation fields or 3D Gaussians, and hybrid methods combining 3D Gaussians and triangular meshes. While the former can reconstruct overall motion, it lacks cross-frame geometric correspondence, resulting in overly smooth and dynamistic surface details in new perspectives. The latter improves stability by introducing triangular meshes, but still struggles to model pose-driven local surface deformations, failing to accurately recover fine-grained details such as wrinkles and muscles in new perspectives. Both types of methods fail to explicitly model the motion-deformation mapping relationship, leading to insufficient generalization ability when dealing with new perspectives and poses, and an inability to effectively decouple surface materials and lighting, limiting the application of the models in diverse rendering scenarios. Summary of the Invention

[0004] This invention addresses the pain point of existing monocular video human body reconstruction methods neglecting surface dynamic deformation modeling. It proposes a method for dynamic reconstruction of human body surface based on monocular video, which solves the technical defects of existing technologies such as insufficient modeling accuracy of local dynamic details such as cloth wrinkles and muscle protrusions, poor geometric consistency under new perspectives, weak generalization ability under new poses, and unrealistic rendering effects under new lighting. It significantly improves the realism and application scope of monocular video human body dynamic reconstruction.

[0005] To achieve the above objectives, the technical solution provided by this invention is as follows: a method for dynamic reconstruction of human body surface based on monocular video. This method is implemented based on a multi-stage progressive human body reconstruction framework, which includes overall human body motion modeling as the first stage, local surface dynamic deformation modeling as the second stage, and surface appearance and lighting modeling as the third stage, specifically as follows: For overall human body motion modeling, a stable and fast three-dimensional human body representation is obtained by using a modeling method based on 3D Gaussian and triangular meshes; for local surface dynamic deformation modeling, it includes: The Vector Quantization Variational Autoencoder (VQ-VAE) is introduced to encode continuous human poses into discrete pose features. New poses can be queried for the nearest discrete pose features through VQ-VAE, thereby improving the generalization ability to unseen poses. A motion-driven surface deformation network is designed to construct a direct mapping relationship from discrete posture features to local surface dynamic deformation. This surface deformation network is based on linear hybrid skin weights. The normalized weights of each 3D Gaussian affected by different joints are modulated by a normalized exponential function softmax. This allows for a more accurate characterization of the correlation between local surface and joint motion under different postures. Subsequently, discrete posture features are used as control signals. The input discrete posture features are weighted and fused based on the above normalized weights. Then, a multilayer perceptron is used to predict the three-dimensional coordinate offset and normal vector offset of each 3D Gaussian. Finally, the triangular mesh is driven to generate a human body surface dynamic mesh synchronized with the posture. The design of the balanced surface regularizer first divides the human body surface into three categories based on wrinkle intensity: no wrinkles, moderate wrinkles, and strong wrinkles. Then, each category is further divided into several local regions. By constraining the average projection of the 3D Gaussian coordinate offset along the normal vector direction within each local region to approach zero, the concavity and convexity of the local regions are balanced, thereby making the dynamic deformation of the surface smooth and avoiding unreasonable geometric abrupt changes. For surface appearance and lighting modeling, based on the Phong lighting model, the differentiable renderer Nvdiffrast is used to decouple the color of the dynamic mesh of the human body surface into surface albedo independent of lighting and ambient lighting parameters related to lighting. Under new lighting conditions, by replacing the ambient lighting parameters and performing physically based rendering on the decoupled surface albedo, a highly realistic appearance image that conforms to the new lighting conditions can be generated.

[0006] The specific implementation of this dynamic reconstruction method for the human body surface includes the following steps:

[0007] 1) Extract keyframes of human motion from monocular video, and simultaneously acquire the RGB image, mask, parameters of SMPL human template, and intrinsic and extrinsic parameters of the keyframes; at the same time, extract the true normal vector map of the corresponding keyframes to construct the basic dataset required for training.

[0008] 2) The basic dataset is fed into the multi-stage progressive human reconstruction framework for training. During the training process, multiple loss functions are jointly optimized and iterated until the loss value converges. The trained multi-stage progressive human reconstruction framework is then used as the optimal reconstruction model.

[0009] 3) Extract new perspective frames of human motion from monocular video, and simultaneously acquire the RGB image of the new perspective frame, the parameters of the SMPL human template, and the intrinsic and extrinsic parameters of the camera; input the RGB image of the new perspective frame, the parameters of the SMPL human template, and the intrinsic and extrinsic parameters of the camera into the optimal reconstruction model obtained in step 2), and the dynamic reconstruction results of the human body under new perspective, new posture and new lighting conditions can be generated, achieving high-quality rendering output with both geometric details and appearance consistency.

[0010] Further, in step 1), the monocular video is sampled at fixed time intervals to extract the RGB images of keyframes, masks, parameters of the SMPL human body template, and intrinsic and extrinsic parameters of the camera. The parameters of the SMPL human body template include shape parameters and pose parameters. The shape parameters are used to initialize the shape of the 3D Gaussian, and the pose parameters are used to describe the motion state of the human joints. The intrinsic parameters of the camera are used to determine the mapping relationship between image pixels and three-dimensional space points, and the extrinsic parameters of the camera are used to characterize the pose of the camera in the world coordinate system. The normal vector map of the corresponding training frame is generated using a normal vector extraction tool as supervision data for the normal vector optimization of the 3D Gaussian.

[0011] Furthermore, the specific implementation of the overall human motion modeling is as follows:

[0012] First, the human body is initialized using the shape parameters of the SMPL human body template. A 3D Gaussian containing attributes such as three-dimensional coordinates, covariance, opacity, color, and normal vector is constructed in the normal space. The 3D Gaussian in the normal space is mapped to the target pose space through linear blending skinning. The 3D Gaussian is then optimized using a joint loss function based on pixel reconstruction error, pixel perception error, and pixel structure similarity to obtain a normalized 3D Gaussian.

[0013] Subsequently, a differentiable Poisson reconstruction and differentiable moving cube algorithm is used to extract triangular meshes from the 3D Gaussian based on the normal vectors of the 3D Gaussian. The triangular meshes are then rendered to obtain a normal vector map. The normal vectors of the 3D Gaussian are optimized by the joint constraints of normal vector reconstruction loss, normal vector perception loss and normal vector structure similarity loss, thereby improving the geometric consistency and detail representation ability of the triangular meshes under multiple perspectives.

[0014] Furthermore, the specific implementation of the local surface dynamic deformation modeling is as follows:

[0015] The first phase has been approved by The first stage consists of a standard 3D Gaussian mesh composed of individual units, which can be converted into a triangular mesh. The second stage aims to establish a mapping relationship between discrete attitude features and local surface dynamic deformation, with its initial input including the three-dimensional coordinate set of the initial 3D Gaussian mesh. Pose parameters of SMPL human body template :

[0016] ;

[0017] ;

[0018] In the formula, The total number of 3D Gaussians. Indicates the first The three-dimensional coordinates of a particle in gauge space There are 23 joints in the human body, excluding the root node. Indicates the first Rotational parameters of each joint;

[0019] To improve the generalization ability of motion-driven surface deformation networks to novel poses not seen in the training set, a pre-trained VQ-VAE was used. Discretization encoding is performed, and the VQ-VAE is encoded by the encoder. Vector quantizer with codebook and decoder constitute;

[0020] First, Convert to continuous rotation representation encoder Will Mapped to continuous pose features at the joint level:

[0021] ;

[0022] In the formula, It is an encoder The output set of continuous pose features. yes For the first Continuous pose features generated by each joint express A set consisting of continuous pose features of each joint;

[0023] Subsequently, By using nearest neighbor search, Independently replace with the closest discrete pose feature in the codebook:

[0024] ;

[0025] In the formula, yes The output set of continuous pose embedding vectors. yes For the first Discrete pose features generated by each joint. express A set consisting of discrete pose features of each joint. yes The output set of discrete attitude features;

[0026] VQ-VAE training employs a joint optimization approach: minimizing the reconstruction loss. With codebook promised loss Achieved, total loss Weighted sum of the two:

[0027] ;

[0028] ;

[0029] In the formula, For discrete attitude feature sets The decoding result, Measuring decoding results and The error between them for The weight coefficients; after training, the VQ-VAE parameters are fixed, and its output... This will serve as the input signal for the subsequent motion-driven surface deformation network;

[0030] To establish the relationship between 3D Gaussians and human joints, enabling each 3D Gaussian to adapt to the influence of the motion of its neighboring joints, a linear blending skin weight is first introduced. Its representative is the first The 3D Gaussian was the first The influence strength of each joint, satisfying To enhance the discriminative power of local joint effects, the weights of this linear hybrid skin were adjusted. Perform softmax processing:

[0031] ;

[0032] In the formula, exp represents the natural exponential function; after processing, we get It is the first The 3D Gaussian was the first Normalized weights affecting the joints; for all 3D Gaussians, they are affected by the first joint. The normalized weights of the influence of each joint are uniformly denoted as... ;

[0033] Subsequently, using For the Discrete pose features of each joint Perform weighted aggregation to generate 3D Gaussian perceptual joint features. :

[0034] ;

[0035] In the formula, It is the first The normalized weights of each joint for each 3D Gaussian, and It is a local pose feature that integrates all joint features;

[0036] The set of three-dimensional coordinates of 3D Gaussian A multi-resolution hash encoding function is used to map the feature to a high-dimensional space to obtain its spatial location characteristics. :

[0037] ;

[0038] In the formula, the multi-resolution hash coding function is denoted as ,pass The set of three-dimensional coordinates of 3D Gaussian By encoding, spatial location features can be obtained. ;

[0039] Next, Input modulation network Learn the scaling parameters of 3D Gaussians and offset parameters ; and then through and Spatial location features Modulation is performed to make Incorporating joint movement information:

[0040] ;

[0041] ;

[0042] In the formula, ⊙ represents element-wise multiplication. This is a spatial location encoding that incorporates joint motion information after modulation. It is a linear layer used to... Mapped to the final feature vector Finally, Input multilayer perceptron Directly predict the deformation offset of the 3D Gaussian under the current attitude drive:

[0043] ;

[0044] In the formula, It is the 3D coordinate offset of a 3D Gaussian, and the deformed 3D coordinate set is: ; It is the normal vector of a 3D Gaussian. It is the offset of the normal vector of the 3D Gaussian vector, and the deformed normal vector is... ;

[0045] Motion-driven surface deformation networks based on pose parameters of SMPL human templates 3D Gaussian coordinate set, linear blending skin weights Predicting the 3D coordinate offset of a 3D Gaussian and normal vector offset The deformed 3D Gaussian is generated; then, the dynamic mesh of the human body surface with dynamic details is extracted from the deformed 3D Gaussian normal vector through the differentiable Poisson reconstruction and differentiable moving cube algorithm.

[0046] When relying solely on monocular video for supervised optimization, motion-driven surface deformation networks tend to produce excessive deformation in the camera's visible area and unreasonable geometric spikes in the invisible area. To constrain the reasonableness and smoothness of the deformation, the human body surface is divided into three regions: wrinkle-free... Moderate folds and strong folds For each region, it is further divided into several local regions composed of adjacent vertices. For each local region, the average projection of the 3D Gaussian coordinate offset along its normal vector direction is calculated.

[0047] ;

[0048] In the formula, Represents the k-th local region. Represents a local area The number of Gaussians in 3D yes The Middle The three-dimensional coordinate offset of a 3D Gaussian. yes The Middle A 3D Gaussian normal vector, Measure the k-th local region The convexity / concavity measurement value, where a positive number represents an overall convexity and a negative number represents an overall concavity;

[0049] By balancing the surface regularizer Encourage all local areas Approaching zero, to achieve a balance between local protrusions and depressions:

[0050] ;

[0051] In the formula, Represents the region type. These are the regularization weight coefficients for the corresponding regions, with the lowest weight for strongly wrinkled regions and the highest weight for unwrinkled regions. It is a region The number of local regions within; the equilibrium surface regularizer It effectively prevents the loss of control over local deformation and ensures the geometric rationality when viewed from different perspectives;

[0052] Total training loss of motion-driven surface deformation networks Depend on and constitute:

[0053] ;

[0054] In the formula, It is the reconstruction loss, used to calculate the error between the normal map obtained by rendering the dynamic mesh of the human body surface and the real normal map. Specifically, it includes normal vector reconstruction loss, normal vector perception loss and normal vector structural similarity loss. It is the regularization loss, used to constrain the stability of local deformations during model training.

[0055] Furthermore, the specific implementation of the surface appearance and lighting modeling is as follows:

[0056] Using the Phong lighting model, the color C of the dynamic mesh on the human body surface is decomposed into surface albedo and ambient lighting parameters. The decomposition formula is as follows:

[0057] ;

[0058] In the formula, The surface albedo of the dynamic mesh on the human body surface is an inherent material property of the mesh, which is only related to the color of the human body surface itself and is independent of lighting conditions; ambient light. Diffuse reflection and specular reflection component These are ambient lighting parameters;

[0059] Diffuse reflection From the normal vector relative to the direction of the light source dot product and light source color Jointly determined:

[0060] ;

[0061] Specular reflection component Color reflected by a mirror Viewpoint and direction Reflection direction and highlight index calculate:

[0062] ;

[0063] ;

[0064] In the formula, It is derived based on the law of vector reflection. It is the dot product of the normal vector and the light source direction, representing the length of the projection of the light source direction onto the normal vector. It is the vector obtained by symmetrically expanding the direction of the light source along the normal vector, minus After that, what was obtained It refers to the direction of the light source after reflection from the surface;

[0065] The surface appearance and lighting modeling employs Nvdiffrast differentiable rendering for parameter optimization. Differentiable rendering refers to gradient-differentiable modeling of the rendering process from vertex to pixel color of the dynamic mesh on the human body surface, using pixel-level error as the loss function to optimize the parameters. , , , , , All of them have gradient information for backpropagation. The surface albedo and ambient lighting parameters are jointly optimized by gradient descent method. Finally, a surface albedo that is bound to the dynamic mesh of the reconstructed human body surface and is independent of lighting is obtained. That is, the human body surface is realistically re-rendered under any new lighting conditions.

[0066] Further, in step 3), firstly, data from the monocular video is read using OpenCV, including the RGB image of the new viewpoint frame, the parameters of the SMPL human body template, and the intrinsic and extrinsic parameters of the camera. This data is then input into the optimal reconstruction model of the multi-stage progressive human body reconstruction framework. The data is processed sequentially through the first and second stages of the optimal reconstruction model, generating a standard 3D Gaussian mesh and a dynamic human body surface mesh containing local surface dynamic deformation, respectively. This dynamic human body surface mesh is then decoupled in the third stage to obtain the surface albedo and ambient lighting parameters. The following processing path is adopted for different application requirements: New perspective rendering requirements: Combine the standard 3D Gaussian and the dynamic mesh of the human body surface with the intrinsic and extrinsic parameters of the camera in the new perspective frame to generate a spatiotemporally consistent geometric rendering result under the new perspective. New pose generalization requirement: Input the pose parameters of the SMPL human template corresponding to the new pose into VQ-VAE, retrieve the nearest discrete pose feature from the codebook of the vector quantizer, and then input the discrete pose feature into the motion-driven surface deformation network to generate local surface dynamic deformation under the new pose. New lighting rendering requirement: Utilize the surface albedo obtained through decoupling to adjust ambient lighting parameters and generate realistic appearance rendering results under different lighting conditions.

[0067] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0068] 1. By combining VQ-VAE, discrete posture features that match human joint movements are retrieved from the codebook to achieve natural and reasonable generation of human surface dynamics under new postures.

[0069] 2. Construct a motion-driven surface deformation network to explicitly establish the mapping relationship between discrete posture features and local surface dynamic deformation, accurately recovering detailed deformations such as clothing folds and muscle movements.

[0070] 3. Introduce a balanced surface regularizer to constrain the dynamic deformation of the surface to maintain a balance between concavity and convexity within the region, ensuring geometric stability under monocular vision.

[0071] 4. Adopting a decoupled modeling scheme of surface albedo and ambient lighting parameters, separating material and lighting information, supporting physically realistic relighting under any new lighting conditions, and expanding the applicable rendering scenarios.

[0072] 5. Based on Nvdiffrast, efficient differentiable mesh rendering is achieved, significantly improving model training efficiency. Attached Figure Description

[0073] Figure 1 This is a schematic diagram of the logic flow of the present invention. In the diagram, multimodal data refers to the RGB image corresponding to the keyframe, the mask, the model parameters of the SMPL human body (including shape parameters and pose parameters), the camera intrinsic and extrinsic parameters, and the real normal vector map.

[0074] Figure 2 This is a flowchart of a motion-driven surface deformation network. In the diagram, Codebook refers to the codebase.

[0075] Figure 3 Training flowchart for modeling surface appearance and lighting. Detailed Implementation

[0076] The present invention will be further described in detail below with reference to the embodiments and accompanying drawings, but the embodiments of the present invention are not limited thereto.

[0077] This embodiment discloses a method for dynamic reconstruction of human body surfaces based on monocular video. This method is implemented based on a multi-stage progressive human body reconstruction framework, which includes: a first stage of overall human motion modeling; a second stage of local surface dynamic deformation modeling; and a third stage of surface appearance and lighting modeling. Specifically: for overall human motion modeling, a stable and rapid 3D human body representation is obtained using a modeling method based on 3D Gaussian and triangular meshes; for local surface dynamic deformation modeling, it includes: The Vector Quantization Variational Autoencoder (VQ-VAE) is introduced to encode continuous human poses into discrete pose features. New poses can be queried for the nearest discrete pose features through VQ-VAE, thereby improving the generalization ability to unseen poses. A motion-driven surface deformation network is designed to construct a direct mapping relationship from discrete posture features to local surface dynamic deformation. This surface deformation network is based on linear hybrid skin weights. The normalized weights of each 3D Gaussian affected by different joints are modulated by a normalized exponential function softmax. This allows for a more accurate characterization of the correlation between local surface and joint motion under different postures. Subsequently, discrete posture features are used as control signals. The input discrete posture features are weighted and fused based on the above normalized weights. Then, a multilayer perceptron is used to predict the three-dimensional coordinate offset and normal vector offset of each 3D Gaussian. Finally, the triangular mesh is driven to generate a human body surface dynamic mesh synchronized with the posture. The design of the balanced surface regularizer first divides the human body surface into three categories based on wrinkle intensity: no wrinkles, moderate wrinkles, and strong wrinkles. Then, each category is further divided into several local regions. By constraining the average projection of the 3D Gaussian coordinate offset along the normal vector direction within each local region to approach zero, the concavity and convexity of the local regions are balanced, thereby making the dynamic deformation of the surface smooth and avoiding unreasonable geometric abrupt changes. For surface appearance and lighting modeling, based on the Phong lighting model, the differentiable renderer Nvdiffrast is used to decouple the color of the dynamic mesh of the human body surface into surface albedo independent of lighting and ambient lighting parameters related to lighting. Under new lighting conditions, by replacing the ambient lighting parameters and performing physically based rendering on the decoupled surface albedo, a highly realistic appearance image that conforms to the new lighting conditions can be generated.

[0078] like Figure 1 As shown, the specific implementation of this dynamic reconstruction method for the human body surface includes the following steps:

[0079] 1) Extract keyframes of human motion from monocular video, and simultaneously acquire the RGB images, masks, parameters of the SMPL human template, and intrinsic and extrinsic parameters of the keyframes; at the same time, extract the ground truth normal vector maps of the corresponding keyframes to construct the basic dataset required for training, as follows:

[0080] The monocular video is sampled at fixed time intervals (e.g., one frame every 5 frames). The RGB images, masks, parameters of the SMPL human template, and intrinsic and extrinsic parameters of the keyframes are extracted. The parameters of the SMPL human template include shape parameters and pose parameters. The shape parameters are used to initialize the shape of the 3D Gaussian, and the pose parameters are used to describe the motion state of the human joints. The intrinsic parameters of the camera are used to determine the mapping relationship between image pixels and points in 3D space, and the extrinsic parameters of the camera are used to characterize the pose of the camera in the world coordinate system. The real normal vector map of the corresponding training frame is generated using a normal vector extraction tool as supervision data for the normal vector optimization of the 3D Gaussian.

[0081] 2) The aforementioned basic dataset is fed into a multi-stage progressive human reconstruction framework for training. The first stage performs overall human motion modeling: firstly, the time-varying 3D Gaussian is optimized to obtain a normalized 3D Gaussian; then, triangular meshes are extracted from the 3D Gaussian using its normal vectors, and the normal vectors are optimized to improve geometric consistency and detail representation. The second stage performs local surface dynamic deformation modeling: firstly, VQ-VAE is introduced to encode continuous human poses into discrete pose features, and then these discrete pose features are input into a motion-driven surface deformation network to learn the discrete pose features and local surface dynamic deformation. The network establishes a mapping relationship between dynamic deformations. Based on linear hybrid skin weights, it modulates the hybrid weights of each 3D Gaussian affected by different joints using a normalized exponential function (softmax). Then, it weights and fuses the discrete pose features of the input based on these hybrid weights. A multilayer perceptron predicts the 3D coordinate offset and normal vector offset corresponding to each 3D Gaussian, ultimately driving the triangular mesh to generate a dynamic human surface mesh synchronized with the pose. Furthermore, a balanced surface regularizer is introduced. The human surface is first divided into three regions based on wrinkle intensity, and then each region is further subdivided into several local regions. Subsequently, the average projection of the 3D Gaussian 3D coordinate offset along the normal vector within each local region is constrained to approach zero, ensuring the generation of a reasonable and stable dynamic human surface mesh. The third stage involves modeling the human surface appearance and lighting: by decoupling the modeling of surface albedo and ambient lighting parameters, it supports realistic re-rendering of the human appearance under different lighting conditions. During training, multiple loss functions are jointly optimized, iterating until the loss value converges, resulting in a well-trained multi-stage progressive human reconstruction framework as the optimal reconstruction model.

[0082] Specifically, the implementation of the overall human motion modeling is as follows:

[0083] First, the human body is initialized using the shape parameters of the SMPL human body template. A 3D Gaussian containing attributes such as three-dimensional coordinates, covariance, opacity, color, and normal vector is constructed in the normal space. The 3D Gaussian in the normal space is mapped to the target pose space through linear blending skinning. The 3D Gaussian is then optimized using a joint loss function based on pixel reconstruction error, pixel perception error, and pixel structure similarity to obtain a normalized 3D Gaussian.

[0084] Subsequently, a differentiable Poisson reconstruction and differentiable moving cube algorithm is used to extract triangular meshes from the 3D Gaussian based on the normal vectors of the 3D Gaussian. The triangular meshes are then rendered to obtain a normal vector map. The normal vectors of the 3D Gaussian are optimized by the joint constraints of normal vector reconstruction loss, normal vector perception loss and normal vector structure similarity loss, thereby improving the geometric consistency and detail representation ability of the triangular meshes under multiple perspectives.

[0085] Specifically, the implementation of the local surface dynamic deformation modeling is as follows:

[0086] The first phase has been approved by The first stage consists of a standard 3D Gaussian mesh composed of individual units, which can be converted into a triangular mesh. The second stage aims to establish a mapping relationship between discrete attitude features and local surface dynamic deformation, with its initial input including the three-dimensional coordinate set of the initial 3D Gaussian mesh. Pose parameters of SMPL human body template :

[0087] ;

[0088] ;

[0089] In the formula, The total number of 3D Gaussians. Indicates the first The three-dimensional coordinates of a particle in gauge space There are 23 joints in the human body, excluding the root node. Indicates the first Rotational parameters of each joint;

[0090] To improve the generalization ability of motion-driven surface deformation networks to novel poses not seen in the training set, a pre-trained VQ-VAE was used. Discretization encoding is performed, and the VQ-VAE is encoded by the encoder. Vector quantizer with codebook and decoder constitute;

[0091] First, Convert to continuous rotation representation encoder Will Mapped to continuous pose features at the joint level:

[0092] ;

[0093] In the formula, It is an encoder The output set of continuous pose features. yes For the first Continuous pose features generated by each joint express A set consisting of continuous pose features of each joint;

[0094] Subsequently, By using nearest neighbor search, Independently replace with the closest discrete pose feature in the codebook:

[0095] ;

[0096] In the formula, yes The output set of continuous pose embedding vectors. yes For the first Discrete pose features generated by each joint. express A set consisting of discrete pose features of each joint. yes The output set of discrete attitude features;

[0097] VQ-VAE training employs a joint optimization approach: minimizing the reconstruction loss. With codebook promised loss Achieved, total loss Weighted sum of the two:

[0098] ;

[0099] ;

[0100] In the formula, For discrete attitude feature sets The decoding result, Measuring decoding results and The error between them for The weight coefficients; after training, the VQ-VAE parameters are fixed, and its output... This will serve as the input signal for the subsequent motion-driven surface deformation network;

[0101] As shown in Figure 2, in order to establish the relationship between 3D Gaussians and human joints, and to enable each 3D Gaussian to adapt to the influence of the motion of its neighboring joints, a linear hybrid skin weight is first introduced. Its representative is the first The 3D Gaussian was the first The influence strength of each joint, satisfying To enhance the discriminative power of local joint effects, the weights of this linear hybrid skin were adjusted. Perform softmax processing:

[0102] ;

[0103] In the formula, exp represents the natural exponential function; after processing, we get It is the first The 3D Gaussian was the first Normalized weights affecting the joints; for all 3D Gaussians, they are affected by the first joint. The normalized weights of the influence of each joint are uniformly denoted as... ;

[0104] Subsequently, using For the Discrete pose features of each joint Perform weighted aggregation to generate 3D Gaussian perceptual joint features. :

[0105] ;

[0106] In the formula, It is the first The normalized weights of each joint for each 3D Gaussian, and It is a local pose feature that integrates all joint features;

[0107] The set of three-dimensional coordinates of 3D Gaussian A multi-resolution hash encoding function is used to map the feature to a high-dimensional space to obtain its spatial location characteristics. :

[0108] ;

[0109] In the formula, the multi-resolution hash coding function is denoted as ,pass The set of three-dimensional coordinates of 3D Gaussian By encoding, spatial location features can be obtained. ;

[0110] Next, Input modulation network Learn the scaling parameters of 3D Gaussians and offset parameters ; and then through and Spatial location features Modulation is performed to make Incorporating joint movement information:

[0111] ;

[0112] ;

[0113] In the formula, ⊙ represents element-wise multiplication. This is a spatial location encoding that incorporates joint motion information after modulation. It is a linear layer used to... Mapped to the final feature vector Finally, Input multilayer perceptron Directly predict the deformation offset of the 3D Gaussian under the current attitude drive:

[0114] ;

[0115] In the formula, It is the 3D coordinate offset of a 3D Gaussian, and the deformed 3D coordinate set is: ; It is the normal vector of a 3D Gaussian. It is the offset of the normal vector of the 3D Gaussian vector, and the deformed normal vector is... ;

[0116] Motion-driven surface deformation networks based on pose parameters of SMPL human templates 3D Gaussian coordinate set, linear blending skin weights Predicting the 3D coordinate offset of a 3D Gaussian and normal vector offset The deformed 3D Gaussian is generated; then, the dynamic mesh of the human body surface with dynamic details is extracted from the deformed 3D Gaussian normal vector through the differentiable Poisson reconstruction and differentiable moving cube algorithm.

[0117] When relying solely on monocular video for supervised optimization, motion-driven surface deformation networks tend to produce excessive deformation in the camera's visible area and unreasonable geometric spikes in the invisible area. To constrain the reasonableness and smoothness of the deformation, the human body surface is divided into three regions: wrinkle-free... Moderate folds and strong folds For each region, it is further divided into several local regions composed of adjacent vertices. For each local region, the average projection of the 3D Gaussian coordinate offset along its normal vector direction is calculated.

[0118] ;

[0119] In the formula, Represents the k-th local region. Represents a local area The number of Gaussians in 3D yes The Middle The three-dimensional coordinate offset of a 3D Gaussian. yes The Middle A 3D Gaussian normal vector, Measure the k-th local region The convexity / concavity measurement value, where a positive number represents an overall convexity and a negative number represents an overall concavity;

[0120] By balancing the surface regularizer Encourage all local areas Approaching zero, to achieve a balance between local protrusions and depressions:

[0121] ;

[0122] In the formula, Represents the region type. These are the regularization weight coefficients for the corresponding regions, with the lowest weight for strongly wrinkled regions and the highest weight for unwrinkled regions. It is a region The number of local regions within; the equilibrium surface regularizer It effectively prevents the loss of control over local deformation and ensures the geometric rationality when viewed from different perspectives;

[0123] Total training loss of motion-driven surface deformation networks Depend on and constitute:

[0124] ;

[0125] In the formula, It is the reconstruction loss, used to calculate the error between the normal map obtained by rendering the dynamic mesh of the human body surface and the real normal map. Specifically, it includes normal vector reconstruction loss, normal vector perception loss and normal vector structural similarity loss. It is the regularization loss, used to constrain the stability of local deformations during model training.

[0126] Specifically, such as Figure 3 As shown, the specific implementation of the surface appearance and lighting modeling is as follows:

[0127] Using the Phong lighting model, the color C of the dynamic mesh on the human body surface is decomposed into surface albedo and ambient lighting parameters. The decomposition formula is as follows:

[0128] ;

[0129] In the formula, The surface albedo of the dynamic mesh on the human body surface is an inherent material property of the mesh, which is only related to the color of the human body surface itself and is independent of lighting conditions; ambient light. Diffuse reflection and specular reflection component These are ambient lighting parameters;

[0130] Diffuse reflection From the normal vector relative to the direction of the light source dot product and light source color Jointly determined:

[0131] ;

[0132] Specular reflection component Color reflected by a mirror Viewpoint and direction Reflection direction and highlight index calculate:

[0133] ;

[0134] ;

[0135] In the formula, It is derived based on the law of vector reflection. It is the dot product of the normal vector and the light source direction, representing the length of the projection of the light source direction onto the normal vector. It is the vector obtained by symmetrically expanding the direction of the light source along the normal vector, minus After that, what was obtained It refers to the direction of the light source after reflection from the surface;

[0136] The surface appearance and lighting modeling employs Nvdiffrast differentiable rendering for parameter optimization. Differentiable rendering refers to gradient-differentiable modeling of the rendering process from vertex to pixel color of the dynamic mesh on the human body surface, using pixel-level error as the loss function to optimize the parameters. , , , , , All of them have gradient information for backpropagation. The surface albedo and ambient lighting parameters are jointly optimized by gradient descent method. Finally, a surface albedo that is bound to the dynamic mesh of the reconstructed human body surface and is independent of lighting is obtained. That is, the human body surface is realistically re-rendered under any new lighting conditions.

[0137] 3) Extract new perspective frames of human motion from monocular video, and simultaneously acquire the RGB image of the new perspective frame, the parameters of the SMPL human template, and the intrinsic and extrinsic parameters of the camera; input the RGB image of the new perspective frame, the parameters of the SMPL human template, and the intrinsic and extrinsic parameters of the camera into the optimal reconstruction model obtained in step 2), and the dynamic reconstruction results of the human body under new perspective, new pose, and new lighting conditions can be generated, achieving high-quality rendering output with both geometric details and appearance consistency, as detailed below:

[0138] First, data from the monocular video is read using OpenCV, including the RGB image of the new viewpoint frame, parameters of the SMPL human body template, and the camera's intrinsic and extrinsic parameters. The new viewpoint frame uses a 30-frame interval sampling strategy. This data is then input into the optimal reconstruction model of the multi-stage progressive human body reconstruction framework. The data is processed sequentially through the first and second stages of the optimal reconstruction model, generating a standard 3D Gaussian mesh and a dynamic human body surface mesh containing local surface dynamic deformations, respectively. This dynamic human body surface mesh is then decoupled in a third stage to obtain surface albedo and ambient lighting parameters. The following processing paths are used to meet different application requirements: New perspective rendering requirements: Combine the standard 3D Gaussian and the dynamic mesh of the human body surface with the intrinsic and extrinsic parameters of the camera in the new perspective frame to generate a spatiotemporally consistent geometric rendering result under the new perspective. New pose generalization requirement: Input the pose parameters of the SMPL human template corresponding to the new pose into VQ-VAE, retrieve the nearest discrete pose feature from the codebook of the vector quantizer, and then input the discrete pose feature into the motion-driven surface deformation network to generate local surface dynamic deformation under the new pose. New lighting rendering requirement: Utilize the surface albedo obtained through decoupling to adjust ambient lighting parameters and generate realistic appearance rendering results under different lighting conditions.

[0139] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A method for dynamic reconstruction of human body surface based on monocular video, characterized in that, This method is based on a multi-stage progressive human body reconstruction framework, which includes three stages: overall human motion modeling (first stage), local surface dynamic deformation modeling (second stage), and surface appearance and lighting modeling (third stage). Specifically: For overall human motion modeling, a stable and rapid 3D human body representation is obtained using a modeling method based on 3D Gaussian and triangular meshes; for local surface dynamic deformation modeling, it includes: The Vector Quantization Variational Autoencoder (VQ-VAE) is introduced to encode continuous human poses into discrete pose features. New poses can be queried for the nearest discrete pose features through VQ-VAE, thereby improving the generalization ability to unseen poses. A motion-driven surface deformation network is designed to construct a direct mapping relationship from discrete posture features to local surface dynamic deformation. This surface deformation network is based on linear hybrid skin weights. The normalized weights of each 3D Gaussian affected by different joints are modulated by a normalized exponential function softmax. This allows for a more accurate characterization of the correlation between local surface and joint motion under different postures. Subsequently, discrete posture features are used as control signals. The input discrete posture features are weighted and fused based on the above normalized weights. Then, a multilayer perceptron is used to predict the three-dimensional coordinate offset and normal vector offset of each 3D Gaussian. Finally, the triangular mesh is driven to generate a human body surface dynamic mesh synchronized with the posture. The design of the balanced surface regularizer first divides the human body surface into three categories based on wrinkle intensity: no wrinkles, moderate wrinkles, and strong wrinkles. Then, each category is further divided into several local regions. By constraining the average projection of the 3D Gaussian coordinate offset along the normal vector direction within each local region to approach zero, the concavity and convexity of the local regions are balanced, thereby making the dynamic deformation of the surface smooth and avoiding unreasonable geometric abrupt changes. For surface appearance and lighting modeling, based on the Phong lighting model, the differentiable renderer Nvdiffrast is used to decouple the color of the dynamic mesh of the human body surface into surface albedo independent of lighting and ambient lighting parameters related to lighting. Under new lighting conditions, by replacing the ambient lighting parameters and performing physically based rendering on the decoupled surface albedo, a highly realistic appearance image that conforms to the new lighting conditions can be generated. The specific implementation of this dynamic reconstruction method for the human body surface includes the following steps: 1) Extract keyframes of human motion from monocular video, and simultaneously acquire the RGB image, mask, parameters of SMPL human template, and intrinsic and extrinsic parameters of the keyframes; at the same time, extract the real normal vector map of the corresponding keyframes to construct the basic dataset required for training. 2) The basic dataset is fed into the multi-stage progressive human reconstruction framework for training. During the training process, multiple loss functions are jointly optimized and iterated until the loss value converges. The trained multi-stage progressive human reconstruction framework is then used as the optimal reconstruction model. 3) Extract new perspective frames of human motion from monocular video, and simultaneously acquire the RGB image of the new perspective frame, the parameters of the SMPL human template, and the intrinsic and extrinsic parameters of the camera; input the RGB image of the new perspective frame, the parameters of the SMPL human template, and the intrinsic and extrinsic parameters of the camera into the optimal reconstruction model obtained in step 2), and the dynamic reconstruction results of the human body under new perspective, new posture and new lighting conditions can be generated, achieving high-quality rendering output with both geometric details and appearance consistency.

2. The method for dynamic reconstruction of human body surface based on monocular video according to claim 1, characterized in that, In step 1), the monocular video is sampled at fixed time intervals to extract the RGB images, mask, parameters of the SMPL human body template, and intrinsic and extrinsic parameters of the keyframes. The parameters of the SMPL human body template include shape parameters and pose parameters. The shape parameters are used to initialize the shape of the 3D Gaussian, and the pose parameters are used to describe the motion state of the human joints. The intrinsic parameters of the camera are used to determine the mapping relationship between image pixels and three-dimensional space points, and the extrinsic parameters of the camera are used to characterize the pose of the camera in the world coordinate system. The normal vector map of the corresponding training frame is generated using a normal vector extraction tool as supervision data for the normal vector optimization of the 3D Gaussian.

3. The method for dynamic reconstruction of human body surface based on monocular video according to claim 1, characterized in that, The specific implementation of the overall human motion model is as follows: First, the human body is initialized using the shape parameters of the SMPL human body template. A 3D Gaussian containing attributes such as three-dimensional coordinates, covariance, opacity, color, and normal vector is constructed in the normal space. The 3D Gaussian in the normal space is mapped to the target pose space through linear blending skinning. The 3D Gaussian is then optimized using a joint loss function based on pixel reconstruction error, pixel perception error, and pixel structure similarity to obtain a normalized 3D Gaussian. Subsequently, a differentiable Poisson reconstruction and differentiable moving cube algorithm is used to extract triangular meshes from the 3D Gaussian based on the normal vectors of the 3D Gaussian. The triangular meshes are then rendered to obtain a normal vector map. The normal vectors of the 3D Gaussian are optimized by the joint constraints of normal vector reconstruction loss, normal vector perception loss and normal vector structure similarity loss, thereby improving the geometric consistency and detail representation ability of the triangular meshes under multiple perspectives.

4. The method for dynamic reconstruction of human body surface based on monocular video according to claim 1, characterized in that, The specific implementation of the local surface dynamic deformation modeling is as follows: The first phase has been approved by The first stage consists of a standard 3D Gaussian mesh composed of individual units, which can be converted into a triangular mesh. The second stage aims to establish a mapping relationship between discrete attitude features and local surface dynamic deformation, with its initial input including the three-dimensional coordinate set of the initial 3D Gaussian mesh. Pose parameters of SMPL human body template : ； ； In the formula, The total number of 3D Gaussians. Indicates the first The three-dimensional coordinates of a particle in gauge space There are 23 joints in the human body, excluding the root node. Indicates the first Rotational parameters of each joint; To improve the generalization ability of motion-driven surface deformation networks to novel poses not seen in the training set, a pre-trained VQ-VAE was used. Discretization encoding is performed, and the VQ-VAE is encoded by the encoder. Vector quantizer with codebook and decoder constitute; First, Convert to continuous rotation representation encoder Will Mapped to continuous pose features at the joint level: ； In the formula, It is an encoder The output set of continuous pose features. yes For the first Continuous pose features generated by each joint express A set consisting of continuous pose features of each joint; Subsequently, By using nearest neighbor search, Independently replace with the closest discrete pose feature in the codebook: ； In the formula, yes The output set of continuous pose embedding vectors. yes For the first Discrete pose features generated by each joint. express A set consisting of discrete pose features of each joint. yes The output set of discrete attitude features; VQ-VAE training employs a joint optimization approach: minimizing the reconstruction loss. With codebook promised loss Achieved, total loss Weighted sum of the two: ； ； In the formula, For discrete attitude feature sets The decoding result, Measuring decoding results and The error between them for The weight coefficients; after training, the VQ-VAE parameters are fixed, and its output... This will serve as the input signal for the subsequent motion-driven surface deformation network; To establish the relationship between 3D Gaussians and human joints, enabling each 3D Gaussian to adapt to the influence of the motion of its neighboring joints, a linear blending skin weight is first introduced. Its representative is the first The 3D Gaussian was the first The influence strength of each joint, satisfying To enhance the discriminative power of local joint effects, the weights of this linear hybrid skin were adjusted. Perform softmax processing: ； In the formula, exp represents the natural exponential function; after processing, we get It is the first The 3D Gaussian was the first Normalized weights affecting the joints; for all 3D Gaussians, they are affected by the first joint. The normalized weights of the influence of each joint are uniformly denoted as... ; Subsequently, using For the first Discrete pose features of each joint Perform weighted aggregation to generate 3D Gaussian perceptual joint features. : ； In the formula, It is the first The normalized weights of each joint for each 3D Gaussian, and It is a local pose feature that integrates all joint features; The set of three-dimensional coordinates of 3D Gaussian A multi-resolution hash encoding function is used to map the feature to a high-dimensional space to obtain the spatial location feature. : ； In the formula, the multi-resolution hash coding function is denoted as ,pass The set of three-dimensional coordinates of 3D Gaussian By encoding, spatial location features can be obtained. ; Next, Input modulation network Learn the scaling parameters of 3D Gaussians and offset parameters ; and then through and Spatial location features Modulation is performed to make Incorporating joint movement information: ； ； In the formula, ⊙ represents element-wise multiplication. This is a spatial location encoding that incorporates joint motion information after modulation. It is a linear layer used to... Mapped to the final feature vector Finally, Input multilayer perceptron Directly predict the deformation offset of the 3D Gaussian under the current attitude: ； In the formula, It is the 3D coordinate offset of a 3D Gaussian, and the deformed 3D coordinate set is... ; It is the normal vector of a 3D Gaussian. It is the offset of the normal vector of the 3D Gaussian vector, and the deformed normal vector is... ; Motion-driven surface deformation networks based on pose parameters of SMPL human templates 3D Gaussian coordinate set, linear blending skin weights Predicting the 3D coordinate offset of a 3D Gaussian and normal vector offset The deformed 3D Gaussian is generated; then, the dynamic mesh of the human body surface with dynamic details is extracted from the deformed 3D Gaussian normal vector through the differentiable Poisson reconstruction and differentiable moving cube algorithm. When relying solely on monocular video for supervised optimization, motion-driven surface deformation networks are prone to excessive deformation in the camera's visible area, while generating unreasonable geometric spikes in the invisible area. To ensure the rationality and smoothness of deformation, the human body surface is divided into three regions: wrinkle-free... Moderate folds and strong folds For each region, it is further divided into several local regions composed of adjacent vertices. For each local region, the average projection of the 3D Gaussian coordinate offset along its normal vector direction is calculated. ； In the formula, Represents the k-th local region. Represents a local area The number of Gaussians in 3D yes The Middle The three-dimensional coordinate offset of a 3D Gaussian. yes The Middle A 3D Gaussian normal vector, Measure the k-th local region The convexity / concavity measurement value, where a positive number represents an overall convexity and a negative number represents an overall concavity; By balancing the surface regularizer Encourage all local areas Approaching zero, to achieve a balance between local protrusions and depressions: ； In the formula, Represents the region type. These are the regularization weight coefficients for the corresponding regions, with the lowest weight for strongly wrinkled regions and the highest weight for unwrinkled regions. It is a region The number of local regions within; the equilibrium surface regularizer It effectively prevents the loss of control over local deformation and ensures the geometric rationality when viewed from different perspectives; Total training loss of motion-driven surface deformation networks Depend on and constitute: ； In the formula, It is the reconstruction loss, used to calculate the error between the normal map obtained by rendering the dynamic mesh of the human body surface and the real normal map. Specifically, it includes normal vector reconstruction loss, normal vector perception loss and normal vector structural similarity loss. It is the regularization loss, used to constrain the stability of local deformations during model training.

5. The method for dynamic reconstruction of human body surface based on monocular video according to claim 1, characterized in that, The specific implementation of the surface appearance and lighting modeling is as follows: Using the Phong lighting model, the color C of the dynamic mesh on the human body surface is decomposed into surface albedo and ambient lighting parameters. The decomposition formula is as follows: ； In the formula, The surface albedo of the dynamic mesh on the human body surface is an inherent material property of the mesh, which is only related to the color of the human body surface itself and is independent of lighting conditions; ambient light. Diffuse reflection and specular reflection component These are ambient lighting parameters; Diffuse reflection From the normal vector relative to the direction of the light source dot product and light source color Jointly determined: ； Specular reflection component Color reflected by a mirror Viewpoint direction Reflection direction and highlight index calculate: ； ； In the formula, It is derived based on the law of vector reflection. It is the dot product of the normal vector and the light source direction, representing the length of the projection of the light source direction onto the normal vector. It is the vector obtained by symmetrically expanding the direction of the light source along the normal vector, minus After that, what was obtained It refers to the direction of the light source after reflection from the surface; The surface appearance and lighting modeling employs Nvdiffrast differentiable rendering for parameter optimization. Differentiable rendering refers to gradient-differentiable modeling of the rendering process from vertex to pixel color of the dynamic mesh on the human body surface, using pixel-level error as the loss function to optimize the parameters. , , , , , All of them have gradient information for backpropagation. The surface albedo and ambient lighting parameters are jointly optimized by gradient descent method. Finally, a surface albedo that is bound to the dynamic mesh of the reconstructed human body surface and is independent of lighting is obtained. That is, the human body surface is realistically re-rendered under any new lighting conditions.

6. The method for dynamic reconstruction of human body surface based on monocular video according to claim 1, characterized in that, In step 3), firstly, data from the monocular video is read using OpenCV, including the RGB image of the new viewpoint frame, the parameters of the SMPL human body template, and the intrinsic and extrinsic parameters of the camera. This data is then input into the optimal reconstruction model of the multi-stage progressive human body reconstruction framework. The data is processed sequentially through the first and second stages of the optimal reconstruction model, generating a standard 3D Gaussian mesh and a dynamic human body surface mesh containing local surface dynamic deformation, respectively. This dynamic human body surface mesh is then decoupled in the third stage to obtain the surface albedo and ambient lighting parameters. The following processing path is adopted for different application requirements: New perspective rendering requirements: Combine the standard 3D Gaussian and the dynamic mesh of the human body surface with the intrinsic and extrinsic parameters of the camera in the new perspective frame to generate a spatiotemporally consistent geometric rendering result under the new perspective. New pose generalization requirement: Input the pose parameters of the SMPL human template corresponding to the new pose into VQ-VAE, retrieve the nearest discrete pose feature from the codebook of the vector quantizer, and then input the discrete pose feature into the motion-driven surface deformation network to generate local surface dynamic deformation under the new pose. New lighting rendering requirement: Utilize the surface albedo obtained through decoupling to adjust ambient lighting parameters and generate realistic appearance rendering results under different lighting conditions.

Images