Drivable physical real digital human simulation method

Abstract

The invention discloses a drivable physical real digital human simulation method, and relates to the technical field of artificial intelligence, and the method comprises the steps: diffusing a skin weight from an SMPL surface to a whole space, and generating a skin weight field; through a linear hybrid skin method, converting a human body from a standard posture to a posture of the human body corresponding to an input frame, and rendering an RGB picture and a depth map of the human body under a new posture by using 2D Gaussian splash; optimization is carried out by minimizing luminosity loss between corresponding frames of a rendered picture and an input RGB picture and a difference between a normal vector diagram calculated by minimizing a depth map and a normal vector diagram predicted by the RGB picture, and self-supervised loss, eccentricity filtering algorithm constraint point cloud distribution and Gaussian ellipse are respectively introduced. Therefore, by adopting the drivable physical real digital human simulation method, a high-fidelity dressing digital human driven by the human body posture sequence and rendered from any view angle can be generated, the time overhead is reduced, and the real-time rendering effect is achieved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a drivable physical real digital human simulation method. Background Art

[0002] Digital humans are complex entities with multiple human characteristics, created through various computer-generated methods and existing in the non-physical world. They have broad application prospects and value in fields such as entertainment, finance, tourism, education, and healthcare. In recent years, digital human dressing technology has made significant progress, particularly in the areas of 3D modeling and virtual try-on.

[0003] Currently, the existing technologies mainly include:

[0004] (1) Patent application number CN202410622381.0 utilizes multi-level detail modeling, texture mapping, and lighting processing methods, utilizing multi-CPU parallel computing technology to improve the rendering efficiency of digital humans in the metaverse while ensuring realism. However, this technology requires the use of 3D modeling software to model the human body and textures, requiring certain professional knowledge. Furthermore, parallel computing on the CPU alone is time-consuming.

[0005] (2) Patent application number CN202410553452.6 extracts 3D features from an input single-person image to determine 3D geometric features; determines 2D pixel features based on the single-person image and the corresponding normal map; extracts a mesh surface based on the 3D geometric features and the 2D pixel features, and reconstructs a 3D digital human using the mesh surface. However, this method requires models such as transformers to extract 3D features, which results in high training time and video memory overhead, and the generated digital human cannot be rendered in real time.

[0006] (3) Patent application number CN202410086329.8 uses human body posture and RGBD image data to construct a shape network and an implicit surface network. The RGBD image is input in real time to generate a three-dimensional point set and extract sampling points. The surface occupancy value is calculated based on the sampling point features, and a three-dimensional human body model is generated using the equal-surface extraction method. Similarly, this technology's specialized RGB-D camera cannot be directly reconstructed using low-cost equipment such as monocular cameras by directly capturing video. The generated implicit surface network is time-consuming and cannot be rendered in real time.

[0007] Therefore, it is necessary to provide a drivable physical real digital human simulation method with low cost, small storage space, low time overhead, and the ability to achieve real-time rendering, while being driven by human posture sequences. Summary of the Invention

[0008] The purpose of the present invention is to provide a drivable physical real digital human simulation method, which can drive the digital human through human posture sequences, while reducing storage space, lowering computing resources, and achieving real-time rendering.

[0009] To achieve the above object, the present invention provides a drivable physical real digital human simulation method, comprising the following steps:

[0010] S1, diffuse the skin weight from the SMPL surface along the normal direction to the entire space to generate the skin weight field;

[0011] S2. Transform the human body from the standard pose to the pose corresponding to the input frame using the linear blending skinning method, and use 2D Gaussian splashing to render the RGB image and depth map of the human body in the new pose;

[0012] S3 is optimized by minimizing the photometric loss between the corresponding frames of the rendered image and the input RGB image, and minimizing the difference between the normal map calculated by the depth map and the normal map estimated by the input RGB image. A self-supervised loss is introduced to constrain the point cloud distribution, and then the eccentricity filtering algorithm is used to eliminate slender Gaussian ellipses.

[0013] Preferably, 2D Gaussian splattering models and reconstructs a geometrically accurate radiation field through multi-view images, uses a 2D ellipse to represent the object, and optimizes the parameters of the Gaussian ellipse by gradient descent.

[0014] Preferably, the Gaussian ellipse has at least one parameter.

[0015] Preferably, the formula for linear blend skinning is as follows:

[0016]

[0017] Where p i is a point on the human body in the standard posture, p' i is the point transformed to the new posture, w k,i Represents the skin weight of the k-th joint to the i-th vertex, G' k represents the affine transformation matrix of the kth joint from the standard pose to the new pose, θ represents the pose parameter, and J(β) represents the position of the joint point.

[0018] Preferably, the self-supervised loss includes area loss and opacity loss;

[0019] Among them, the area loss is used to constrain the uniform distribution of ellipses, and the opacity loss constrains the distribution of point clouds on the surface of the digital human.

[0020] Therefore, the present invention adopts the above-mentioned drivable physical real digital human simulation method, which has the following technical effects:

[0021] (1) Using multi-view videos taken by a monocular camera for training can reduce costs. At the same time, the training process adopts the simplest gradient descent method without using complex networks such as convolutional neural networks, which reduces the training time overhead and reduces computing resources.

[0022] (2) A 2D Gaussian point cloud is used to explicitly represent the clothed digital human, which reduces storage space and uses the Gaussian splash rendering algorithm to achieve real-time rendering; in addition, the generated digital human can also be driven by a human posture sequence.

[0023] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 It is a flow chart of a drivable physically realistic digital human simulation method. DETAILED DESCRIPTION

[0025] The present invention can be explained in more detail by the following examples. The purpose of disclosing the present invention is to protect all changes and improvements within the scope of the present invention. The present invention is not limited to the following examples.

[0026] like Figure 1 As shown, the present invention provides a drivable physically realistic digital human simulation method, the goal of which is to create a drivable physically realistic digital human based on a given multi-view monocular video and the SMPL parameters corresponding to each human body frame in each video. The method specifically includes the following steps:

[0027] S1. Diffuse the skin weights from the SMPL surface to the entire space to generate a skin weight field. By querying the skin weight field, the skin weights of each Gaussian ellipse can be obtained. SMPL is a parametric human body model that generates and manipulates 3D human models by controlling a small number of parameters (such as pose and shape).

[0028] S2. Takes multi-view monocular video as input and transforms the human body from a standard pose to the pose corresponding to the input frame through linear blending skinning. 2D Gaussian splashing is then used to render the RGB image and depth map of the human body in the new pose. Training time is also reduced, enabling fast training, and the real-time frame rate can reach around 60FPS. At the same time, the geometric representation capability is enhanced, enabling real-time rendering of the appearance. Compared with triangular meshes, there is no need to obtain the corresponding texture map, and the appearance can be rendered directly, ultimately generating a high-fidelity clothed digital human that can be driven by a human pose sequence and rendered from any perspective. The human pose sequence can be obtained through methods such as motion capture, artist design, visual estimation, or AIGC generation.

[0029] Among them, the formula of linear blend skinning is generally:

[0030]

[0031] Where p i is a point on the human body in the standard posture, p' i is the point transformed to the new posture, w k,i Represents the skin weight of the k-th joint to the i-th vertex, G' k represents the affine transformation matrix of the kth joint from the standard pose to the new pose, θ represents the pose parameter, and J(β) represents the position of the joint point.

[0032] 2D Gaussian splatting is a new method for modeling and reconstructing geometrically accurate radiation fields from multi-view images. Objects are represented using explicit 2D ellipses. Each Gaussian ellipse can be trained with multiple parameters, which are then optimized via gradient descent. Compared to traditional implicit radiation field methods, Gaussian splatting renders images similarly to rasterization, eliminating sampling time and enabling real-time rendering while also accelerating training.

[0033] S3 is optimized by minimizing the photometric loss between the rendered image and the corresponding frame of the input RGB image, and minimizing the difference between the normal vector map calculated from the depth map and the normal vector map estimated from the input RGB image.

[0034] Furthermore, linear blend skinning is used to drive triangular meshes, and these meshes are evenly distributed across the geometric surface. To be compatible with linear blend skinning, the Gaussian point cloud should also be evenly distributed across the surface to avoid artifacts like clothing clipping when driving the human body.

[0035] However, Gaussian ellipses are typically not evenly distributed; high-frequency regions are always more abundant, while low-frequency regions are rare. Therefore, during the optimization process, a self-supervised loss is introduced to constrain the distribution of the Gaussian ellipses and the smoothness of the normal map. This is used to constrain the distribution of the point cloud, ensuring that it is evenly distributed on the surface of the human body. When driving the digital human, each point cloud is assigned a skinning weight. The self-supervised loss includes an area loss and an opacity loss. The area loss constrains the area of the ellipse, ensuring that all ellipses are evenly distributed. The opacity loss ensures that the opacity of each ellipse is as close to 1 or 0 as possible, thereby ensuring that the ellipses are distributed on the surface of the digital human.

[0036] In addition, since the linear blend skin weights are diffused from SMPL, which are themselves used for triangular meshes, a self-supervised loss (area loss) is used to constrain the scaling of each Gaussian ellipse and minimize the variance of the product of the two semi-axis vectors of the ellipse. This makes the Gaussian ellipse as evenly distributed as the face of the triangular mesh.

[0037] Furthermore, the area variance of triangular mesh patches is small, so overly elongated patches are almost nonexistent. While area loss only constrains the area of the Gaussian ellipse, some very elongated ellipses may still exist, potentially leading to uneven geometric edges and burrs. Based on this, this embodiment also employs an eccentricity filtering algorithm. By eliminating Gaussian ellipses with eccentricities exceeding a threshold, this algorithm uses the eccentricity of the ellipse to adaptively control density, thereby removing overly elongated Gaussian ellipses.

[0038] Therefore, the present invention adopts the above-mentioned drivable physical real digital human simulation method. The generated digital human can be driven by a human body posture sequence, so that the human body posture conforms to the physical laws, the folds of clothes are more realistic, the rendering speed is improved, and real-time performance is met.

[0039] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit the same. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solutions of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A method for simulating a drivable physical real digital human, characterized in that: The following steps are involved: S1, diffuse the skin weight from the SMPL surface along the normal direction to the entire space to generate the skin weight field; S2. Transform the human body from the standard pose to the pose corresponding to the input frame using the linear blending skinning method, and use 2D Gaussian splashing to render the RGB image and depth map of the human body in the new pose; S3 is optimized by minimizing the photometric loss between the corresponding frames of the rendered image and the input RGB image, and minimizing the difference between the normal map calculated by the depth map and the normal map estimated by the input RGB image. A self-supervised loss is introduced to constrain the point cloud distribution, and then the eccentricity filtering algorithm is used to eliminate slender Gaussian ellipses.

2. A drivable physical real digital human simulation method according to claim 1, characterized in that: 2D Gaussian splatting models and reconstructs geometrically accurate radiation fields through multi-view images, uses 2D ellipses to represent objects, and optimizes the parameters of the Gaussian ellipse by gradient descent.

3. A drivable physical real digital human simulation method according to claim 2, characterized in that: The Gaussian ellipse has at least one parameter.

4. The method for simulating a drivable physical real digital human according to claim 1, characterized in that: The formula for linear blend skinning is as follows: Where p i is a point on the human body in the standard posture, p' i is the point transformed to the new posture, w k,i Represents the skin weight of the k-th joint to the i-th vertex, G' k represents the affine transformation matrix of the kth joint from the standard pose to the new pose, θ represents the pose parameter, and J(β) represents the position of the joint point.

5. The method for simulating a drivable physical real digital human according to claim 1, characterized in that: Self-supervised losses include area loss and opacity loss; Among them, the area loss is used to constrain the uniform distribution of ellipses, and the opacity loss constrains the distribution of point clouds on the surface of the digital human.

Images