Human body reconstruction and real-time animation rendering system based on 3D Gaussian

Abstract

The present invention belongs to the field of computer graphics technology, specifically a 3D Gaussian-based human body reconstruction and real-time animation rendering system, including a method, and the specific steps are as follows: parameter extraction, deformable Gaussian human body initialization, setting learnable and optimized parameters, deformable Gaussian human body deformation, rendering, model training, real-time human body animation. The present invention does not require complex equipment, only needs to capture all positions of the human body, and train the data for about 3 minutes, and can effectively reconstruct an accurate human body from a monocular video. At the same time, the present invention can output a human body image sequence at a speed of 120fps to achieve realistic human body animation. The present invention is more widely used in the field of dynamic human body modeling and real-time rendering, especially in the fields of virtual reality, game development and animation production, providing a more efficient and realistic solution.

Description

Technical Field

[0001] The invention relates to the technical field of computer graphics, in particular to a 3D Gaussian-based human body reconstruction and real-time animation rendering system. Background Art

[0002] In the field of digital human modeling, although traditional three-dimensional models are widely used in virtual reality, film and animation production, the creation of these models usually relies on cumbersome manual operations and motion capture technology, resulting in high costs and labor intensity. Parametric 3D human models simplify equipment and labor costs, but their fixed topological structures show obvious limitations when simulating complex details such as clothing and hairstyles. In addition, in recent years, human modeling technology based on neural radiance fields (NeRF) can use deep learning to accurately capture details, but such methods are sensitive to deformation and have low rendering efficiency, especially at high resolutions. This also limits its application in real-time animation.

[0003] The directness, flexibility and efficiency of 3D Gaussian and the rapidity of Gaussian splash rendering technology make 3D Gaussian modeling technology an ideal choice for human body modeling. However, when 3D Gaussian modeling technology is applied to human body models, there is no loose clothing for modeling, there are errors in human body posture parameters and LBS weights, and there is a lack of Gaussian optimization strategies specifically for human body models. Therefore, a human body reconstruction and real-time animation rendering system based on 3D Gaussian is invented. Summary of the invention

[0004] In view of the above problems and / or the problems existing in the existing 3D Gaussian-based human body reconstruction and real-time animation rendering systems, the present invention is proposed.

[0005] Therefore, the purpose of the present invention is to provide a 3D Gaussian-based human body reconstruction and real-time animation rendering system that can solve the above-mentioned existing problems.

[0006] To solve the above technical problems, according to one aspect of the present invention, the present invention provides the following technical solutions:

[0007] A human body reconstruction and real-time animation rendering system based on 3D Gaussian includes a method, and the specific steps are as follows:

[0008] S1, parameter extraction: Use deep learning methods to identify and track key points of the human body, such as head, arms, legs, etc., from each frame of the monocular video, extract human morphological parameters and posture parameters, and extract the parameters of the shooting camera to build a data set;

[0009] S2, deformable Gaussian human body initialization: With the help of the template prior provided by SMPL, the various parameters of Gaussian are initialized according to different human body parts to obtain a deformable Gaussian human body;

[0010] S3, setting parameters that can be learned and optimized: setting human body morphology parameters and posture parameters, linear mixed skin parameters in SMPL, and Gaussian parameters of deformable Gaussian human body as parameters that can be learned and optimized, adding learnable mixed skin correction parameters to correct linear mixed skin parameters, and adding learnable linear mixed skin parameters of potential joints;

[0011] S4, deformable Gaussian human body deformation: according to the posture parameters, the deformable Gaussian human body is deformed by using the linear blend skinning technology and the potential joint module to obtain a Gaussian human body with posture;

[0012] S5, rendering: using a differentiable 3D Gaussian sputtering rendering method, rendering a 2D image of a Gaussian human body with pose;

[0013] S6, model training: The 2D image obtained by S5 is compared with the real image to find the gradient to optimize the parameters, and the number of Gaussians in the S5 deformable Gaussian human body is optimized using the innovative compact densification algorithm and KL-Sph re-Gaussianization algorithm;

[0014] S7, real-time human body animation: input the new custom posture parameter sequence into the trained model, output the new human posture in real time, and realize animation.

[0015] As a preferred solution of the 3D Gaussian-based human body reconstruction and real-time animation rendering system of the present invention, the specific process of S2 is as follows:

[0016] S21, Gaussian mean initialization: human body morphological parameters are input into the SMPL model, and the mesh vertex coordinates matching the human body in the video are output, and each vertex coordinate is used as the Gaussian mean;

[0017] S22, initialization of other Gaussian parameters: use the normal vector calculated from the human body mesh and two random orthogonal vectors as the initial rotation parameters of the Gaussian, set a smaller Gaussian scaling parameter in the detail-dense areas of the face, hands and feet, set the Gaussian opacity to 1, and set the Gaussian spherical harmonic coefficient to zero.

[0018] As a preferred solution of the 3D Gaussian-based human body reconstruction and real-time animation rendering system of the present invention, the specific process of S4 is as follows:

[0019] S41, basic posture transformation: the posture parameters and the linear blend skinning parameters are used to calculate the basic transformation matrix of the human body mesh vertices;

[0020] S42, simulating the transformation of clothing behavior: inputting the posture parameters and time parameters into the potential joint module implemented by the multi-layer perceptron, obtaining the posture information of the potential joints, and then calculating the vertex transformation matrix simulating the clothing behavior with the additionally introduced linear mixed skinning parameters;

[0021] S43, linearly adding the two transformation matrices to obtain a final vertex transformation matrix and applying the matrix to the deformable Gaussian human body to obtain the Gaussian human body with posture.

[0022] As a preferred solution of the 3D Gaussian-based human body reconstruction and real-time animation rendering system of the present invention, the specific process of S5 is as follows:

[0023] S51: projecting the Gaussian in the Gaussian human body with the pose to the image plane according to the perspective direction and the camera parameters;

[0024] S52: Calculate the color contribution value generated by all Gaussians at each pixel on the image plane to obtain a 2D image of the model.

[0025] As a preferred solution of the 3D Gaussian-based human body reconstruction and real-time animation rendering system of the present invention, the specific process of S6 is as follows:

[0026] S61: Compact densification algorithm calculates the distance between Gaussians and their nearest neighbor Gaussians and adds new Gaussians between them to adjust the density;

[0027] S62: The KL-Sph re-Gaussian algorithm uses the difference between the KL divergence and the spherical harmonic function to select the Gaussians that need to be split or cloned, avoiding the invalid splitting and cloning of redundant Gaussians. In addition, the loss function is used to optimize and adjust the parameters in the deformable Gaussian human body model.

[0028] As a preferred solution of the 3D Gaussian-based human body reconstruction and real-time animation rendering system of the present invention, the specific process of S7 is as follows:

[0029] S71: The new human body posture parameters and linear mixed skin parameters calculate the basic posture transformation matrix. The new human body posture parameters are input into the clothing behavior transformation matrix output by the potential joint module. The two transformation matrices are linearly added and acted on the deformable Gaussian human body.

[0030] S72: Rendering a 2D image of the deformed Gaussian human body model according to the perspective direction and the camera parameters, wherein the process renders the image in real time at 120 fps to achieve a real-time rendering effect.

[0031] Compared with existing technologies:

[0032] The present invention does not require complicated equipment. It only needs to capture all positions of the human body and train the data for about 3 minutes. It can effectively reconstruct an accurate human body from a monocular video. At the same time, the present invention can output a human body image sequence at a speed of 120fps to achieve realistic human body animation. The present invention has a wider application in the field of dynamic human body modeling and real-time rendering, especially in the fields of virtual reality, game development and animation production, providing a more efficient and realistic solution. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 It is the work flow chart of the present invention;

[0034] Figure 2 A schematic diagram of Gaussian human joints of the present invention;

[0035] Figure 3 This is the output effect diagram of the model of the present invention. DETAILED DESCRIPTION

[0036] To make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0037] The present invention provides a human body reconstruction and real-time animation rendering system based on 3D Gaussian. Figure 1-Figure 3 , including a method, the specific steps are as follows:

[0038] S1, parameter extraction: Use a monocular camera G to shoot a video of the human body H. The human body opens its arms and slowly rotates 360° in place to obtain complete body information. Use deep learning methods to obtain morphological parameters representing b features of the human body Attitude parameters Camera internal parameters Camera extrinsics The posture parameters are specifically the rotation parameters of each joint relative to the parent joint. Figure 2 It is a schematic diagram of the joint;

[0039] S2, deformable Gaussian human body initialization: input β into SMPL and compare it with the surface vertices of the human body mesh template M already in SMPL And parameters Compute a human mesh that matches the body shape of the individual in the video Vertex in Set the vertex coordinates to the Gaussian of the deformable Gaussian body The mean from Calculate the normal vector for each point And randomly generate two Orthogonal vectors Rotation of the Gaussian Scaling parameters Set the opacity to a smaller value of 0.001 in detail-dense areas such as the face, hands, and feet, and 0.01 for the rest; Spherical harmonic coefficients are uniformly set to 1; Set to 0; the deformable Gaussian human body is represented by

[0040] S3, set the parameters that can be learned and optimized: human body shape parameters β and posture parameters θ i , linear blend skinning parameters in SMPL Gaussian parameters of the deformable Gaussian body Set to parameters that can be learned and optimized; i Additional learnable hybrid skinning correction parameters And w i Corrected to Set additional learnable linear blend skinning parameters for z potential joints

[0041] S4, Deformable Gaussian Human Deformation: Joint Regression Parameters Using SMPL Calculate the joint position of M in Use the joint transformation function R in SMPL to calculate the posture parameter θ i , that is, the transformation matrix corresponding to the rotation of the joint axis angle Then calculate the basic transformation matrix of the human body mesh vertices Then, θ i And time parameter t is input into the potential joint module L of the multilayer perceptron to obtain the transformation matrix of the potential joint in γ(t) is based on Fourier transform, With additional designed learnable linear blend skinning parameters Calculate the additional transformation matrix of the human body mesh vertices Finally, the two transformation matrices are linearly added to obtain the final transformation matrix Will Gaussian parameters acting on a deformable Gaussian body Get the Gaussian of the Gaussian body with pose in The Gaussian body with pose is represented as

[0042] S5, rendering: according to the camera intrinsic parameters Camera extrinsics And the perspective projection function π(x; E, K), which transforms the Gaussian human body with posture Projected onto the image plane, Gauss The corresponding 2D mean variance A point on the image plane The volume density contribution is Refer to the volume density formula to render the color value of the point Where d is the camera focus point Direction, is f i Encoded spherical harmonics; the color value of a pixel on the image plane is obtained by sampling the color values ​​of several points;

[0043] S6, model training: the obtained 2D image is compared with the real image to obtain the gradient loss to optimize the learnable parameters; then the compact densification algorithm is used to calculate the average distance between each Gaussian and its k nearest neighbors. For sufficiently dense areas such as face, hands and feet, no densification is performed, while KD-Tree is used to identify other parts and add new Gaussians when necessary; then, the KL-SphRe-Gaussian algorithm is used to reduce the generation of redundant Gaussians by calculating the KL divergence and the difference of spherical harmonic functions, optimize the overall model performance, and improve the accuracy and efficiency of Gaussian distribution; the training is iterated for 3000 cycles;

[0044] S7, real-time human body animation: input the new custom posture parameter sequence into the trained model, output the new human posture in real time, and realize animation.

[0045] The "linear skinning technology" mentioned in the present invention is a technology that converts skeleton motion into mesh vertex deformation in three-dimensional animation, and uses weighted linear combinations of bone joints to achieve smooth movement of vertices.

[0046] The “potential joint module” mentioned in the present invention is a network structure based on a multi-layer perceptron, with a network depth of 2 layers and a width of 256; the input is γ(t) and θ1, θ2, ..., θ l , the output dimensions are the rotation quaternions of the 4 potential joints and the 3D translation transformation.

[0047] The “spherical harmonic function difference” mentioned in the present invention is used to evaluate the similarity between a Gaussian and its neighboring Gaussians. i The difference between the spherical harmonic function and the adjacent Gaussian is calculated as in Yes i The spherical harmonic coefficients The jth element of Yes i Spherical harmonic coefficients of neighboring Gaussians The j-th element of .

[0048] The “KL divergence” mentioned in this invention is the Kullback-Leibler divergence, which is used to evaluate the similarity between two distributions. i With the neighboring Gauss The KL divergence difference between is calculated as:

[0049]

[0050] The code for “setting learnable parameters” mentioned in the present invention is written in Python and implemented based on the Pytorch framework.

[0051] The “deep learning algorithm” mentioned in this invention refers to the DeepCap model.

[0052] Although the present invention has been described above with reference to the embodiments, various modifications may be made thereto and parts thereof may be replaced by equivalents without departing from the scope of the present invention. In particular, as long as there is no structural conflict, the various features in the embodiments disclosed in the present invention may be used in combination with each other in any manner, and the fact that these combinations are not exhaustively described in this specification is only for the sake of omitting space and saving resources. Therefore, the present invention is not limited to the specific embodiments disclosed herein, but includes all technical solutions falling within the scope of the claims.

Claims

1. A 3D Gaussian-based human body reconstruction and real-time animation rendering system, characterized in that: A method is included, and the specific steps are as follows: S1, parameter extraction: Use deep learning methods to identify and track the key points of the human body from each frame of the monocular video, extract the human body shape parameters and posture parameters, and extract the parameters of the shooting camera to build a data set; S2, initialization of deformable Gaussian human body: with the help of the template prior provided by SMPL, each parameter of Gaussian is initialized according to different parts of the human body to obtain a deformable Gaussian human body; the specific process of S2 is as follows: S21, initialization of Gaussian mean: the human body morphological parameters are input into the SMPL model, and the mesh vertex coordinates of the human body in the matching video are output, and each vertex coordinate is used as the mean of Gaussian; S22, initialization of other parameters of Gaussian: the normal vector and two random orthogonal vectors calculated from the human body mesh are used as the initial rotation parameters of Gaussian, and the Gaussian scaling parameter with a value of 0.001 is set in the detail-dense areas of the face, hands and feet, the Gaussian opacity is uniformly set to 1, and the spherical harmonic coefficient of Gaussian is set to zero; S3, set the parameters that can be learned and optimized: human body morphology parameters and posture parameters, linear mixed skin parameters in SMPL, Gaussian parameters of deformable Gaussian body Set as the parameters that can be learned and optimized, where is the mean of the Gaussian, is the Gaussian rotation parameter, is the scaling parameter of Gaussian, is the Gaussian opacity parameter, Spherical harmonic coefficients of Gaussian, add learnable hybrid skin correction parameters to correct linear hybrid skin parameters, add learnable linear hybrid skin parameters for potential joints; S4, deformation of the deformable Gaussian human body: according to the posture parameters, the deformable Gaussian human body is deformed by using the linear blend skinning technology and the potential joint module to obtain a Gaussian human body with posture; the specific process of S4 is as follows: S41, basic pose transformation: use the pose parameters and linear blend skinning parameters to calculate the basic transformation matrix of the human body mesh vertices; S42, simulating the transformation of clothing behavior: inputting the posture parameters and time parameters into a potential joint module implemented by a multi-layer perceptron, obtaining the posture information of the potential joints, and then calculating the vertex transformation matrix simulating the clothing behavior with the learnable linear mixed skinning parameters of the potential joints; The output dimensions of the potential joint module are the rotation quaternions and 3D translation transformations of the 4 potential joints; S43, linearly adding the two transformation matrices to obtain a final vertex transformation matrix and applying the matrix to the deformable Gaussian human body to obtain the Gaussian human body with posture; S5, rendering: using a differentiable 3D Gaussian sputtering rendering method, rendering a 2D image of a Gaussian human body with pose; S6, model training: The 2D image obtained in S5 is compared with the real image to calculate the gradient to optimize the parameters, and the number of Gaussians in the deformable Gaussian human body in S5 is optimized using the compact densification algorithm and the KL-Sph re-Gaussianization algorithm; The specific process of S6 is as follows: S61: Compact densification algorithm calculates the distance between a Gaussian and its nearest neighbor Gaussians and adds new Gaussians between them to adjust the density; S62: KL-Sph re-Gaussian algorithm uses the difference between KL divergence and spherical harmonic function to select Gaussians that need to be split or cloned, and uses loss function to optimize and adjust Gaussian parameters of deformable Gaussian human body; S7, real-time human body animation: input the new custom posture parameter sequence into the trained model, output the new human posture in real time, and realize animation.

2. The 3D Gaussian-based human body reconstruction and real-time animation rendering system according to claim 1, characterized in that: The specific process of S5 is as follows: S51: projecting the Gaussian in the Gaussian human body with the pose to the image plane according to the perspective direction and the camera parameters; S52: Calculate the color contribution value generated by all Gaussians at each pixel on the image plane to obtain a 2D image of the model.

3. The 3D Gaussian-based human body reconstruction and real-time animation rendering system according to claim 1, characterized in that: The specific process of S7 is as follows: S71: Use the new human body posture parameters and linear mixed skin parameters to calculate the basic posture transformation matrix, input the new human body posture parameters into the potential joint module to output the clothing behavior transformation matrix, and the two transformation matrices are linearly added and applied to the deformable Gaussian human body; S72: Rendering a 2D image of the deformed Gaussian human body model according to the perspective direction and the camera parameters, wherein the process renders the image in real time at 120 fps.

Images