Dynamic human body modeling method based on three-dimensional Gaussian sputtering technology

Abstract

A high-quality digital human avatar is quickly reconstructed from a monocular video and is rendered in real time, the method is widely applied to various fields, and an existing method based on a neural radiation field is high in training and rendering cost. According to the dynamic human body three-dimensional modeling method based on the three-dimensional Gaussian sputtering technology, three-dimensional Gaussian can be expanded to a dynamic human scene, and rapid training and real-time rendering can be achieved. Specifically, the three-dimensional Gaussian of the human body is initialized in a standard space, and the three-dimensional Gaussian is converted from the standard space to a posture space by utilizing linear hybrid skin (LBS) according to posture parameters, so that fine details of the human body are learned at very low calculation cost; a posture parameter refining module and an LBS weight optimization module are designed to improve the problem of inaccurate detection of posture parameters in a real data set, and multi-resolution hash coding is introduced to accelerate the model training speed.

Description

Technical Field

[0001] The invention relates to the fields of computer graphics and computer vision, and in particular to a dynamic human body reconstruction and real-time rendering technology based on three-dimensional Gaussian sputtering technology. Background Art

[0002] In traditional 3D human modeling and rendering technologies, it is usually necessary to rely on multi-view stereo vision systems or depth sensors to obtain the depth information of the scene. These methods are often costly, complex equipment and inconvenient for real-time application. With the development of deep learning technology, 3D human reconstruction technology based on monocular video has made significant progress, but these methods still face challenges when dealing with dynamic human scenes, especially in real-time rendering and detail capture.

[0003] Existing deep learning-based 3D human reconstruction methods mainly rely on convolutional neural networks (CNNs) to predict the 3D structure of the human body from 2D images. Although these methods have achieved some success in static scenes, in dynamic scenes, due to the rapid changes in human posture and appearance, these methods often have difficulty in accurately capturing the details of the human body, especially in the modeling of non-rigid parts such as clothing and hair.

[0004] As an emerging 3D scene representation method, NeRF technology achieves the ability to synthesize high-quality images from sparse viewpoints by learning the continuous volume density and color field of the scene. However, NeRF technology has limitations when processing dynamic scenes, mainly because it requires iterative optimization of the multi-layer perceptron (MLP), which makes the training and rendering process slow and difficult to meet the needs of real-time rendering.

[0005] In order to improve rendering efficiency, researchers have proposed a variety of acceleration techniques. For example, Fridovich et al. proposed using sparse geometric expressions to exclude sampling areas that do not contribute to the integral and reduce the number of samples. Reiser et al. proposed using voxel grids to store high-dimensional features or lightweight networks to achieve low-complexity queries. Muller et al. proposed establishing multi-scale voxel grids to store high-dimensional features and compressing high-resolution grids with hashes, which can achieve high resolution and fast rendering under low-complexity conditions. These methods have improved the rendering speed to a certain extent, but the adaptability and detail capture capabilities in dynamic scenes still need to be improved, and these methods still require high time and memory costs.

[0006] In addition, existing methods based on human parametric models, such as the dynamic human body modeling method proposed by Peng et al., use the human body parametric model SMPL as a priori to establish large-scale skeleton motion connections between frames, and simultaneously optimize the non-rigid deformation field and NeRF under standard postures. Although it can accurately control the three-dimensional mesh of the human body through a set of parameters and deform the human body in the observation space into the standard space for rendering through the deformation field, SMPL can only parameterize the naked human body and has limitations in reconstructing details such as hair and clothing with non-rigid deformations. Summary of the invention

[0007] This paper proposes a new method that can quickly complete high-quality human reconstruction and real-time rendering from monocular video. Our method consists of a human body parameterized model SMPL model initialized in a three-dimensional Gaussian in the canonical space and a Gaussian optimization field with multi-resolution hash coding, and designs an LBS weight optimization field and a pose refinement field to solve the problem of inaccurate pose parameters estimated from the image. Then, the LBS transformation is used to transform the three-dimensional Gaussian from the canonical space to the pose space, and the fine parts of the human body are fitted through the Gaussian refinement field, and finally a high-quality reconstruction result of the human body is achieved from a randomly captured monocular video.

[0008] A dynamic human body modeling method based on three-dimensional Gaussian sputtering technology of the present invention comprises the following steps:

[0009] S1: Normalize the human body prior parameter model initialization;

[0010] S2: Linear blending skinning algorithm based on 3D Gaussian;

[0011] S3: pose refinement field, linear blend skinning parameter field, and 3D Gaussian parameter refinement field;

[0012] S4: Model training.

[0013] Furthermore, the specific steps of S1 are:

[0014] S11: The quality of the initialization of the 3D Gaussian in the canonical space has a significant impact on the convergence speed of the model and the quality of the rendering results. Previous studies have used the COLMAP technique to initialize the 3D Gaussian from sparse point clouds or random point clouds generated by Structure from Motion, but these methods mainly perform well in static scenes. However, these methods perform poorly in single-view and cannot estimate point clouds of dynamic data, making them unsuitable for effective and fast human modeling.

[0015] To solve this problem, in our model, we use the human body prior points of the SMPL model to initialize the 3D Gaussian. Specifically, for the SMPL model, we randomly sample 20 additional points in the area of ​​each vertex and copy the vertex skinning parameters to the corresponding sampled points. Then, we initialize the SMPL mesh vertices as the 3D Gaussian centers and create a model containing approximately 137,800 3D Gaussian points. We allow the Gaussian to deviate from the mesh to simulate details such as hair and clothing. In the subsequent optimization process, we trim the 3D Gaussian away from the SMPL mesh. This initialization strategy fully considers the structural characteristics of the human body.

[0016] S12: SMPL (Skinned Multi-Person Linear model) is a skeleton-driven parametric human model M(β, θ), which contains a human template mesh in a static pose. The model has n v Vertices, K joints. Given the human body shape parameters and pose parameters SMPL transforms the template grid into The vertices in are transformed to canonical space:

[0017]

[0018] in, and are the vertex coordinate offsets based on shape and pose, respectively. is the vertex position in the canonical space, and the vertex offset B is caused by the average human shape T plus the morphological blend shape S S Vertex offset B caused by pose blend shape P P The mesh vertex coordinates after the transformation are more consistent with the body shape characteristics of the human body in a static posture. In order to transform the human body mesh into the target posture space, SMPL applies the linear blend skinning (LBS) algorithm to transform the i-th vertex in the canonical space Transform to pose space:

[0019]

[0020] in, represents the mixed weight of the kth joint point to the i-th vertex, G k (J,θ) is the transformation matrix of the kth joint relative to the root node, and J includes K joint positions.

[0021] Furthermore, the specific steps of S2 are:

[0022] S21: We use a differentiable Gaussian rasterization pipeline to represent a 3D scene with a set of unstructured discrete 3D Gaussians. A 3D Gaussian is defined by its center μ in world space and a 3D covariance matrix ∑. The 3D Gaussian can be projected into 2D for rendering, and the 2D covariance matrix ∑′ is calculated as:

[0023] ∑′=PW∑W T P T

[0024] Where P is the Jacobian matrix of the affine approximation of the projective transformation, and W is the view transformation matrix from the world coordinate system to the camera coordinate system. To simplify the optimization process, the covariance matrix ∑ is decomposed into two independently optimized elements, namely the rotation matrix R and the scaling matrix S to describe the shape of the 3D Gaussian:

[0025] ∑=RSS T R T

[0026] Each Gaussian is paired with a spherical harmonic (SH) to represent the directional appearance component. We then perform depth sorting on the projected Gaussians that overlap each pixel and compute their color c i and opacity α i Finally, the color of a pixel is calculated by alpha blending the N ordered Gaussians in that pixel:

[0027]

[0028] S22: Traditional neural radiation field-based methods require numerous ray sampling and inverse linear blending skinning deformation in the observation space, which has training efficiency issues in high-resolution scenes and does not meet the requirements of real-time rendering.

[0029] We adopt a novel approach to optimize the explicit Gaussian distribution in the canonical space and dynamically transform it into the pose space. This approach greatly improves the rendering efficiency and meets the real-time rendering requirements of dynamic scenes. Specifically, given a 3D Gaussian position x in the canonical space, c , the mixed weight w of the kth joint to the vertex k , the transformation matrix G of the kth joint k (J,θ), the transformed Gaussian position is:

[0030]

[0031] Among them, x p To standardize the three-dimensional Gaussian position in space, K is the number of joints. At the same time, in order to ensure the anisotropic characteristics of the three-dimensional Gaussian, we also transform the covariance matrix of each three-dimensional Gaussian:

[0032] Σ p =G(J,θ)Σ c G(J,θ) T

[0033] Among them, Σ c ,∑ p are the covariance matrices of the three-dimensional Gaussian in the normative space and pose space, respectively, is the rotation matrix.

[0034] Furthermore, the specific steps of S3 are:

[0035] S31: We use the most advanced video human pose and shape estimation methods to predict the SMPL body model parameters for each frame in the video, but experiments show that the results predicted by these methods are different from the true values, especially in terms of the accuracy of depth information, which will affect the convergence speed and rendering quality of the model. To solve this problem, we introduced a dedicated pose parameter optimization module MLP pose (·), this module aims to learn the relative changes of joint angles and thus improve the accuracy of parameter estimation.

[0036]

[0037] Among them, θ SMPL are the SMPL human posture parameters estimated from the image.

[0038] In the LBS transformation formula, w k is the mixing weight of the kth joint defined in the canonical space. If we directly use the original weight to perform LBS transformation, serious errors will occur. To solve this problem, we initialize the mixing weight of each 3D Gaussian vertex in the canonical space according to the SMPL model and use a single MLP network MLP LBS (x c )→Δw k Predict the corresponding weight bias. For the i-th vertex, the vertex mixed weight calculation formula is:

[0039]

[0040] Among them, w k,i Expressed as the mixed weight of the k-th joint point to the i-th vertex, we define

[0041]

[0042] S32: Dynamic scene rendering cannot rely solely on vertex color information, because the 3D Gaussian is unstable in the early stage of LBS transformation, and more samples and iterations are required to achieve model convergence. In addition, the 3D Gaussian rasterization technology can only optimize a limited number of Gaussians per iteration, making the model convergence process slow.

[0043] To solve this problem, we move the vertex x c The input is fed into the MLP to calculate the spherical harmonic coefficients (SH). This method allows us to optimize all adjacent Gaussians simultaneously in a single iteration. At the same time, we use the vertex offset estimated by the MLP to adjust the vertex position of each frame and adjust the local details of the human body to further improve accuracy and consistency. We define the parameter field as follows:

[0044] δ x ,SH=MLP Θ (x c ,θ)

[0045] However, the biggest problem with MLP is its low efficiency. In order to reconstruct the scene with high quality, the model often requires a relatively large MLP network, and it takes a lot of time for each sampling point to pass through the network once.

[0046] To solve this problem, we are inspired by Instant-NGP and perform a three-dimensional Gaussian point position x in the standard space. c Perform multi-resolution hash coding, that is, y = enc (x c ;ψ).

[0047] This encoding can improve the approximation quality and training speed in a wide range of applications without incurring significant performance overhead. Specifically, the encoding parameter ψ is divided into L resolution levels, each level contains T feature vectors of dimension F, each level is independent, and we store the feature vectors at the vertices of the grid. For a given input coordinate x c , we find the surrounding voxel grid at resolution level l and assign hash table indices to its grid corners. For all obtained corner indices, we extract the indices from the hash table t l The F-dimensional feature vector corresponding to each angle index can be found in . Then, we c Linear interpolation of the feature vectors at the relative position within the voxel grid at that level is used to obtain the feature vector of the vertex at that level. Finally, the feature vectors of each level are combined to generate the MLP input We input y into a small MLP to calculate the spherical harmonics coefficients (SH) and vertex offsets, and we modify the Gaussian parameter refinement field definition to:

[0048] δ x ,SH=MLPΘ (enc(x p ;ψ),θ)

[0049] At the same time, we also adjusted the LBS weight formula:

[0050]

[0051] Since our 3D Gaussian model is initialized based on the SMPL model, the center of the 3D Gaussian is evenly distributed only in the vicinity of the human body surface. Therefore, in the parameter field, we only need to sample at a fixed position close to the human body surface, rather than in the entire 3D space. This method significantly reduces the storage requirements of the hash table. Thanks to this,

[0052] We adopt multi-resolution hash coding to construct the parameter field to reduce time and storage consumption.

[0053] Furthermore, the specific steps of S4 are:

[0054] S41: Our method uses single-view camera parameters, human body parameter model SMPL parameters, human body mask and image as input. During the training process, a frame of image is randomly selected and the 3D Gaussian is projected onto the image plane for rendering. Since our proposed method can accurately fit dynamic human scenes, no additional regularization loss is required. Therefore, we use L1 loss and S3IM loss to compare the rendered image with the real image. The loss function is expressed as

[0055]

[0056] Among them, λ is the loss weight, we set λ1=0.8, λ2=1.0. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 Flowchart of this method DETAILED DESCRIPTION

[0058] In order to facilitate those skilled in the art to understand and implement the present invention, the present invention is further described in detail below with reference to examples and drawings.

[0059] As attached Figure 1 As shown, a dynamic human body modeling method based on three-dimensional Gaussian sputtering technology includes the following steps:

[0060] Step 1: Normalize the human body prior parameter model initialization

[0061] SMPL (Skinned Multi-Person Linear model) is a skeleton-driven parametric human model M(β, θ), which contains a human template mesh in a static pose. The model has n v Vertices, K joints. Given the human body shape parameters and pose parameters SMPL transforms the template grid into The vertices in are transformed to canonical space:

[0062]

[0063] in, and are the vertex coordinate offsets based on shape and pose, respectively. is the vertex position in the canonical space, and the vertex offset B is caused by the average human shape T plus the morphological blend shape S S Vertex offset B caused by pose blend shape P P The mesh vertex coordinates after the transformation are more consistent with the body shape characteristics of the human body in a static posture. In order to transform the human body mesh into the target posture space, SMPL applies the linear blend skinning (LBS) algorithm to transform the i-th vertex in the canonical space Transform to pose space:

[0064]

[0065] in, represents the mixed weight of the kth joint point to the i-th vertex, G k (J,θ) is the transformation matrix of the kth joint relative to the root node, and J includes K joint positions.

[0066] The quality of the initialization of the 3D Gaussian in the canonical space has a significant impact on the convergence speed of the model and the quality of the rendering results. Previous studies have used COLMAP technology to initialize 3D Gaussians from sparse point clouds or random point clouds generated by Structure from Motion, but these methods mainly perform well in static scenes. However, these methods perform poorly in single-view and cannot estimate point clouds of dynamic data, making them unsuitable for effective and fast human modeling.

[0067] To solve this problem, in our model, we use the human body prior points of the SMPL model to initialize the 3D Gaussian. Specifically, for the SMPL model, we randomly sample 20 additional points in the area of ​​each vertex and copy the vertex skinning parameters to the corresponding sampled points. Then, we initialize the SMPL mesh vertices as the 3D Gaussian centers and create a model containing approximately 137,800 3D Gaussian points. We allow the Gaussian to deviate from the mesh to simulate details such as hair and clothing. In the subsequent optimization process, we trim the 3D Gaussian away from the SMPL mesh. This initialization strategy fully considers the structural characteristics of the human body.

[0068] Step 2: 3D Gaussian-based linear blending skinning algorithm

[0069] We use a differentiable Gaussian rasterization pipeline to represent a 3D scene with a set of unstructured discrete 3D Gaussians. A 3D Gaussian is defined by its center μ in world space and a 3D covariance matrix Σ. The 3D Gaussian can be projected into 2D for rendering, and the 2D covariance matrix Σ′ is calculated as:

[0070] Σ′=PWΣW T P T

[0071] Where P is the Jacobian matrix of the affine approximation of the projective transformation, and W is the view transformation matrix from the world coordinate system to the camera coordinate system. To simplify the optimization process, the covariance matrix ∑ is decomposed into two independently optimized elements, namely the rotation matrix R and the scaling matrix S to describe the shape of the 3D Gaussian:

[0072] ∑=RSS T R T

[0073] Each Gaussian is paired with a spherical harmonic (SH) to represent the directional appearance component. We then perform depth sorting on the projected Gaussians that overlap each pixel and compute their color c i and opacity α i Finally, the color of a pixel is calculated by alpha blending the N ordered Gaussians in that pixel:

[0074]

[0075] Traditional neural radiation field-based methods require numerous ray sampling and inverse linear blending skinning deformation in the observation space, but have training efficiency issues in high-resolution scenes and do not meet the requirements of real-time rendering.

[0076] We adopt a novel approach to optimize the explicit Gaussian distribution in the canonical space and dynamically transform it into the pose space. This approach greatly improves the rendering efficiency and meets the real-time rendering requirements of dynamic scenes. Specifically, given a 3D Gaussian position x in the canonical space, c , the mixed weight w of the kth joint to the vertex k , the transformation matrix G of the kth joint k (J,θ), the transformed Gaussian position is:

[0077]

[0078] Among them, x p To standardize the three-dimensional Gaussian position in space, K is the number of joints. At the same time, in order to ensure the anisotropic characteristics of the three-dimensional Gaussian, we also transform the covariance matrix of each three-dimensional Gaussian:

[0079] Σ p =G(J,θ)Σ c G(J,θ) T

[0080] Among them, ∑ c ,∑ p are the covariance matrices of the three-dimensional Gaussian in the normative space and pose space, respectively, is the rotation matrix.

[0081] Step 3: Pose refinement field, linear blend skinning parameter field, and 3D Gaussian parameter refinement field

[0082] We use the most advanced video human pose and shape estimation methods to predict the SMPL body model parameters for each frame in the video, but experiments show that the results predicted by these methods are different from the true values, especially in terms of the accuracy of depth information, which will affect the convergence speed and rendering quality of the model. To solve this problem, we introduce a dedicated pose parameter optimization module MLP pose (·), this module aims to learn the relative changes of joint angles and thus improve the accuracy of parameter estimation.

[0083]

[0084] Among them, θ SMPL are the SMPL human posture parameters estimated from the image.

[0085] In the LBS transformation formula, w kis the mixing weight of the kth joint defined in the canonical space. If we directly use the original weight to perform LBS transformation, serious errors will occur. To solve this problem, we initialize the mixing weight of each 3D Gaussian vertex in the canonical space according to the SMPL model and use a single MLP network MLP LBS (x c )→Δw k Predict the corresponding weight bias. For the i-th vertex, the vertex mixed weight calculation formula is:

[0086]

[0087] Among them, w k,i Expressed as the mixed weight of the k-th joint point to the i-th vertex, we define

[0088]

[0089] Dynamic scene rendering cannot rely solely on vertex color information, because the 3D Gaussian is unstable in the early stage of LBS transformation, and more samples and iterations are required to achieve model convergence. In addition, the 3D Gaussian rasterization technology can only optimize a limited number of Gaussians per iteration, making the model convergence process slow.

[0090] To solve this problem, we move the vertex x c The input is fed into the MLP to calculate the spherical harmonic coefficients (SH). This method allows us to optimize all adjacent Gaussians simultaneously in a single iteration. At the same time, we use the vertex offset estimated by the MLP to adjust the vertex position of each frame and adjust the local details of the human body to further improve accuracy and consistency. We define the parameter field as follows:

[0091] δ x ,SH=MLP Θ (x c ,θ)

[0092] However, the biggest problem with MLP is its low efficiency. In order to reconstruct the scene with high quality, the model often requires a relatively large MLP network, and it takes a lot of time for each sampling point to pass through the network once.

[0093] To solve this problem, we calculate the three-dimensional Gaussian point position x in the standard space c Perform multi-resolution hash coding, that is, y = enc (x c ; ψ). This encoding can improve the approximation quality and training speed in a wide range of applications without incurring significant performance overhead. Specifically, the encoding parameter ψ is divided into L resolution levels, each level contains T feature vectors of dimension F, each level is independent, and we store the feature vectors at the vertices of the grid. For a given input coordinate xc , we find the surrounding voxel grid at resolution level l and assign hash table indices to its grid corners. For all obtained corner indices, we extract the indices from the hash table t l The F-dimensional feature vector corresponding to each angle index can be found in . Then, we c Linear interpolation of the feature vectors at the relative position within the voxel grid at that level is used to obtain the feature vector of the vertex at that level. Finally, the feature vectors of each level are combined to generate the MLP input We input y into a small MLP to calculate the spherical harmonics coefficients (SH) and vertex offsets, and we modify the Gaussian parameter refinement field definition to:

[0094] δ x ,SH=MLP Θ (enc(x p ;ψ),θ)

[0095] At the same time, we also adjusted the LBS weight formula:

[0096]

[0097] Since our 3D Gaussian model is initialized based on the SMPL model, the center of the 3D Gaussian is evenly distributed only in the vicinity of the human body surface. Therefore, in the parameter field, we only need to sample at a fixed position close to the human body surface, rather than in the entire 3D space. This method significantly reduces the storage requirements of the hash table. To this end, we use multi-resolution hash coding to construct the parameter field to reduce time and storage consumption.

[0098] Step 4: Model training

[0099] Our method uses single-view camera parameters, human body parameter model SMPL parameters, human body mask and image as input. During the training process, a frame of image is randomly selected and the 3D Gaussian is projected onto the image plane for rendering. Since our proposed method can accurately fit dynamic human scenes, no additional regularization loss is required. Therefore, we use L1 loss and S3IM loss to compare the rendered image with the real image. The loss function is expressed as

[0100]

[0101] Among them, λ is the loss weight, we set λ1=0.8, λ2=1.0.

Claims

1. A dynamic human body modeling method based on three-dimensional Gaussian sputtering technology, characterized in that The following steps are involved: S1: Normalize the human body prior parameter model initialization; S2: Linear blending skinning algorithm based on 3D Gaussian; S3: pose refinement field, linear blend skinning parameter field, and 3D Gaussian parameter refinement field; S4: Model training.

2. A dynamic human body modeling method based on three-dimensional Gaussian sputtering technology according to claim 1, characterized in that The specific method of step S1 is: S11: In our model, we use the human body prior points of the SMPL model to initialize the 3D Gaussian. Specifically, for the SMPL model, we randomly sample 20 additional points in the area of ​​each vertex and copy the vertex skinning parameters to the corresponding sampling points. Then, we initialize the SMPL mesh vertices as the 3D Gaussian centers and create a 3D Gaussian grid containing approximately 137 points. For the model with 800 3D Gaussian points, we allow the Gaussian to deviate from the grid to simulate details such as hair and clothing. In the subsequent optimization process, we trim the 3D Gaussian away from the SMPL grid. This initialization strategy fully considers the structural characteristics of the human body; S12: SMPL (Skinned Multi-Person Linear model) is a skeleton-driven parametric human model M(β, θ), which contains a human template mesh in a static pose. The model has n v vertices, K joints, given human body shape parameters and pose parameters SMPL transforms the template grid into The vertices in are transformed to canonical space: in, and are the vertex coordinate offsets based on shape and pose, respectively. is the vertex position in the canonical space, and the vertex offset B caused by the average human shape T plus the morphological blend shape S S Vertex offset B caused by pose blend shape P P The mesh vertex coordinates after the target pose are more consistent with the body shape characteristics in the static pose. In order to transform the human body mesh into the target pose space, SMPL applies the linear blend skinning (LBS) algorithm to transform the i-th vertex in the canonical space. Transform to pose space: in, represents the mixed weight of the kth joint point to the i-th vertex, G k (J, θ) is the transformation matrix of the kth joint relative to the root node, and J includes K joint positions.

3. The method for dynamic human body modeling based on three-dimensional Gaussian sputtering technology according to claim 1, characterized in that The specific steps in step S2 are: S21: We use a differentiable Gaussian rasterization pipeline to represent a 3D scene with a set of unstructured discrete 3D Gaussians. The 3D Gaussians are defined by their center μ in world space and the 3D covariance matrix ∑. The 3D Gaussians can be projected into 2D for rendering. The calculation formula of the 2D covariance matrix ∑′ is: ∑′=PW∑W T P T Where P is the Jacobian matrix of the affine approximation of the projective transformation, W is the view transformation matrix from the world coordinate system to the camera coordinate system, and in order to simplify the optimization process, the covariance matrix ∑ is decomposed into two independently optimized elements, namely the rotation matrix R and the scaling matrix S to describe the shape of the three-dimensional Gaussian: ∑=RSS T R T Each Gaussian is paired with a spherical harmonic (SH) to represent the directional appearance component, after which we deep sort the projected Gaussians that overlap each pixel and compute their color c i and opacity α i ; Finally, the color of a pixel is calculated by alpha blending the N ordered Gaussians in that pixel: S22: Traditional neural radiation field-based methods require a large number of ray sampling and inverse linear blending skin deformation in the observation space, which has training efficiency problems in high-resolution scenes and does not meet the requirements of real-time rendering; We adopt a novel approach, which is to optimize the explicit Gaussian distribution in the canonical space and dynamically transform it into the pose space. This processing method greatly improves the rendering efficiency and meets the real-time rendering requirements of dynamic scenes. Specifically, given the 3D Gaussian position x in the canonical space, c , the mixed weight w of the kth joint to the vertex k , the transformation matrix G of the kth joint k (J, θ), the transformed Gaussian position is: Among them, x p To standardize the three-dimensional Gaussian position in space, K is the number of joints. At the same time, in order to ensure the anisotropic characteristics of the three-dimensional Gaussian, We also transform the covariance matrix of each 3D Gaussian: ∑ p =G(J,θ)∑ c G(J,θ) T Among them, ∑ c ,∑ p are the covariance matrices of the three-dimensional Gaussian in the normative space and pose space, respectively, is the rotation matrix.

4. The method for dynamic human body modeling based on three-dimensional Gaussian sputtering technology according to claim 1, characterized in that The specific steps in step S3 are: S31: We use the most advanced video human pose and shape estimation methods to predict the SMPL body model parameters for each frame in the video, but experiments show that the results predicted by these methods are different from the true values, especially in terms of the accuracy of depth information, which will affect the convergence speed and rendering quality of the model. To solve this problem, we introduced a dedicated pose parameter optimization module MLP pose (·), this module aims to learn the relative changes of joint angles to improve the accuracy of parameter estimation; Among them, θ SMPL It is the SMPL human posture parameter estimated from the image; In the LBS transformation formula, w k is the mixed weight of the kth joint defined in the canonical space. If we directly use the original weight to perform LBS transformation, serious errors will occur. To solve this problem, we initialize the mixed weight of each three-dimensional Gaussian vertex in the canonical space according to the SMPL model and use a single MLP network MLP LBS (x c )→Δw k Predict the corresponding weight bias. For the i-th vertex, the vertex mixed weight calculation formula is: Among them, w k,i Expressed as the mixed weight of the k-th joint point to the i-th vertex, we define S32: Dynamic scene rendering cannot rely solely on vertex color information, because the 3D Gaussian is unstable in the early stage of LBS transformation, and more samples and iterations are required to achieve model convergence. In addition, the 3D Gaussian rasterization technology can only optimize a limited number of Gaussians per iteration, which makes the model convergence process slow; To solve this problem, we move the vertex x c The input is fed into the MLP to calculate the spherical harmonic coefficients (SH). This method allows us to optimize all adjacent Gaussians simultaneously in a single iteration. At the same time, we use the vertex offset estimated by the MLP to adjust the vertex position of each frame and adjust the local details of the human body to further improve accuracy and consistency. We define the parameter field as follows: δ x ,SH=MLP Θ (x c ,θ) However, the biggest problem with MLP is its low efficiency. In order to reconstruct the scene with high quality, the model often requires a relatively large MLP network. It takes a lot of time for each sampling point to pass through the network once. To solve this problem, we are inspired by Instant-NGP and perform a three-dimensional Gaussian point position x in the standard space. c Perform multi-resolution hash coding, that is, y = enc (x c ψ), this encoding can improve the approximation quality and training speed in a wide range of applications without incurring significant performance overhead. Specifically, the encoding parameter ψ is divided into L resolution levels, each level contains T feature vectors of dimension F, and each level is independent. We store the feature vectors at the vertices of the grid. For a given input coordinate x c , we find the voxel grid around it at the resolution level of level, and assign hash table indexes to its grid corners. For all the obtained corner indices, we can find the F-dimensional feature vector corresponding to each corner index from the hash table t, and then, we can calculate the F-dimensional feature vector according to x c Linear interpolation of the feature vectors at the relative position within the voxel grid at that level is used to obtain the feature vector of the vertex at that level. Finally, the feature vectors of each level are combined to generate the MLP input We input y into a small MLP to calculate the spherical harmonics coefficients (SH) and vertex offsets, and we modify the Gaussian parameter refinement field definition to: d x ,SH=MLP Θ (enc(x p ;ψ),θ) At the same time, we also adjusted the LBS weight formula: Since our 3D Gaussian model is initialized based on the SMPL model, the center of the 3D Gaussian is evenly distributed only in the vicinity of the human body surface. Therefore, in the parameter field, we only need to sample at a fixed position close to the human body surface. This method significantly reduces the storage requirements of the hash table without having to do it in the entire three-dimensional space. Thanks to this, we use multi-resolution hash coding to construct the parameter field to reduce time and storage consumption.

5. The method for dynamic human body modeling based on three-dimensional Gaussian sputtering technology according to claim 1, characterized in that The specific steps in step S4 are: S41: Our method uses single-view camera parameters, human body parameter model SMPL parameters, human body mask and image as input. During the training process, a frame of image is randomly selected and the 3D Gaussian is projected onto the image plane for rendering. Since our proposed method can accurately fit dynamic human scenes, no additional regularization loss is required. Therefore, we use L1 loss and S3IM loss to compare the rendered image and the real image. The loss function is expressed as Among them, λ is the loss weight, we set λ1=0.8, λ2=1.0.

Images