A method for generating high-precision dynamic digital humans

Abstract

This invention discloses a method for generating high-precision dynamic digital humans in the field of computer vision. The method includes: extracting skeletal motion and frame-level latent codes from multi-view videos to drive a coarse-model network to generate a 3D mesh, and combining dual quaternion skinning and embedded deformation to achieve pose alignment; subsequently, obtaining the 3D correspondence of vertices through 2D point tracking and depth maps to optimize mesh alignment accuracy; training an animated Gaussian texture on this basis to achieve texture detail modeling; further introducing a physics solver to solve the physical response of the cloth based on the skeletal pose and material type to eliminate clipping and distortion; simultaneously using a spatiotemporal Transformer to model texture offsets and improve dynamic consistency; and finally, completing high-fidelity image rendering through analytical sputtering to generate a detailed, stable, and controllable dynamic digital human.

Description

Technical Field

[0001] This invention belongs to the field of computer vision, and in particular relates to a method for generating high-precision dynamic digital humans. Background Technology

[0002] Digital humans, a key research area in computer graphics and vision, aim to create realistic, animable, and controllable full-body 3D models. These models accurately map skeletal movement to geometric deformation and surface appearance changes, achieving natural and fluid human motion. With the rapid development of applications such as virtual reality (VR), augmented reality (AR), film and television special effects, gaming, virtual customer service, and online education, the demand for high-quality digital humans is increasing. Especially in fields like the metaverse, e-commerce live streaming, remote conferencing, and virtual idols, digital humans have become a core element in building immersive interactive experiences. A high-fidelity, dynamically controllable, and richly detailed digital human not only enhances the realism and immersion of the user experience but also plays a crucial role in professional scenarios such as medical simulation, behavioral analysis, and intelligent manufacturing.

[0003] While neural rendering methods based on implicit representations have made significant progress in digital human modeling in recent years, particularly in generating animable avatars and half-body digital humans, these methods still have significant limitations. First, most existing methods struggle to preserve the highest level of detail, such as high-frequency features like clothing textures, skin wrinkles, and hair structures, especially noticeable when zoomed in by a virtual camera or rendered at 4K or higher resolutions. Second, insufficient accuracy in tracking surface geometry during training, especially in cases of depth estimation errors and drift between character geometry and the real surface, forces appearance modeling to compensate for these geometric errors, further impacting the final visual quality. Furthermore, many methods rely on fixed templates for registration, failing to effectively handle the challenges posed by complex clothing, dynamic deformations, and individual differences, limiting their performance in terms of diversity and generalization ability. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention provides a method for generating high-precision dynamic digital humans, comprising:

[0005] Each frame of the video sequence and its corresponding 3D skeleton pose are obtained from the multi-view video sequence.

[0006] Frame-level skeletal motion features are extracted based on the three-dimensional skeletal pose.

[0007] A learnable latent code is generated based on the skeletal motion features and frame index;

[0008] The latent code and skeletal motion features are input into the coarse model network to obtain the three-dimensional vertices of the coarse model mesh for each frame.

[0009] Based on the coarse model mesh, geometric deformation is performed by combining double quaternion skinning and embedded deformation methods to obtain the deformed coarse model mesh.

[0010] Based on the deformed coarse mesh, image rendering is performed and 2D point tracking is performed in conjunction with the real image to obtain the correspondence between vertices in the image space.

[0011] The optimal corresponding point of each vertex in the three-dimensional space is obtained based on the 2D correspondence and the depth map;

[0012] Based on the corresponding 3D points, a vertex-level alignment loss function is constructed to optimize the position of the vertices in the coarse mesh.

[0013] Based on the optimized vertex positions, construct the texture element-level correspondence and establish a loss function to obtain an animated Gaussian texture;

[0014] Based on the Gaussian texture, it is rendered into the image space using the analytical Gaussian sputtering method to generate a high-resolution dynamic digital human image.

[0015] Preferably, the process of extracting frame-level skeletal motion features based on the three-dimensional skeletal pose includes: normalizing the joint translation of the three-dimensional skeletal pose, and performing motion modeling on multiple frames of images using a sliding window.

[0016] Preferably, the process of generating a learnable latent code based on the skeletal motion features and frame index includes: the latent code is encoded by the frame index and skeletal motion features through a positional encoding function, and then input into a shallow multilayer perceptron to generate a model of geometric random variations that cannot be explained by skeletal motion.

[0017] Preferably, the coarse model network outputs a set of three-dimensional coarse model vertices in the standard pose based on the input skeletal motion features and latent codes. The coarse model network includes a graph convolutional embedding deformation module and a vertex translation neural network.

[0018] Preferably, the dual quaternion skinning technique is used to deform the mesh under the standard pose into the mesh under the target pose, and the embedded deformation is used to predict the vertex translation vector and the adjacency structure through a graph convolutional neural network.

[0019] Preferably, the rendered image and the real image are compared using a two-dimensional point tracker to establish the drift of vertices in the image space, and the three-dimensional points are projected into two-dimensional image points using camera parameters;

[0020] The optimal corresponding point in three dimensions is obtained by bilinear interpolation of the depth map, combined with the view. Figure 1 The consistency score selects the most reliable 3D point from multiple camera views as the final corresponding point;

[0021] The vertex-level alignment loss includes the predicted Euclidean distance between the vertex and the optimal corresponding 3D point, and the loss function is used in conjunction with the Chamfe distance for mesh optimization.

[0022] Preferably, it also includes: inputting the optimized coarse mesh vertices into the cloth physics calculation module, constructing the energy function of the cloth system based on the physical parameters generated by the skeleton pose, material type and latent code, and solving the vertex positions of the cloth after deformation based on the energy function, which is used to correct the three-dimensional geometry of the dynamic clothing;

[0023] The energy function includes tensile energy, bending energy, and shear energy. The solution process is based on the implicit Euler method, which uses the conjugate gradient method to minimize the objective function containing the mass matrix, elastic potential energy, and predicted inertia term, and obtains the equilibrium vertex displacement that satisfies the physical constraints.

[0024] Preferably, it also includes: based on the obtained initial Gaussian texture parameters, further constructing a spatiotemporal residual offset based on inter-frame motion features and appearance features, and using a spatiotemporal Transformer module to perform local attention modeling in the UV space to obtain the texture element offset adjustment amount between consecutive frames;

[0025] The spatiotemporal Transformer module constructs neighborhood constraints using UV radius and temporal frame difference, and fuses motion features with Gaussian texture features through a multi-head attention mechanism to output residual offsets used to correct the temporal consistency of texture elements.

[0026] On the other hand, the present invention also provides an electronic device including a memory, a processor, and a computing program stored in the memory and executable on the processor, wherein the processor implements the method when executing the computing program.

[0027] On the other hand, the present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method.

[0028] Compared with the prior art, the present invention has the following advantages and technical effects:

[0029] At the geometric level, the collaboration between depth alignment and physical calculators eliminates the penetration and distortion in the joint bending region of traditional methods; at the dynamic detail level, the spatiotemporal residual field achieves continuous modeling of wrinkle evolution at 4K resolution for the first time, keeping the fabric microstructure stable when viewed under magnification; at the system efficiency level, the sparse attention mechanism and GPU parallelization design ensure the real-time performance of ultra-high-definition rendering. Attached Figure Description

[0030] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0031] Figure 1 This is a schematic diagram of the method flow according to an embodiment of the present invention. Detailed Implementation

[0032] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0033] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0034] Example 1

[0035] like Figure 1 As shown, this embodiment provides a method for generating high-precision dynamic digital humans, including:

[0036] Each frame of the video sequence and its corresponding 3D skeleton pose are obtained from the multi-view video sequence.

[0037] Frame-level skeletal motion features are extracted based on the three-dimensional skeletal pose.

[0038] A learnable latent code is generated based on the skeletal motion features and frame index;

[0039] The latent code and skeletal motion features are input into the coarse model network to obtain the three-dimensional vertices of the coarse model mesh for each frame.

[0040] Based on the coarse model mesh, geometric deformation is performed by combining double quaternion skinning and embedded deformation methods to obtain the deformed coarse model mesh.

[0041] Based on the deformed coarse mesh, image rendering is performed and 2D point tracking is performed in conjunction with the real image to obtain the correspondence between vertices in the image space.

[0042] The optimal corresponding point of each vertex in the three-dimensional space is obtained based on the 2D correspondence and the depth map;

[0043] Based on the corresponding 3D points, a vertex-level alignment loss function is constructed to optimize the position of the vertices in the coarse mesh.

[0044] Based on the optimized vertex positions, construct the texture element-level correspondence and establish a loss function to obtain an animated Gaussian texture;

[0045] Based on the Gaussian texture, it is rendered into the image space using the analytical Gaussian sputtering method to generate a high-resolution dynamic digital human image.

[0046] The input data is a set of multi-view video sequences. Where f represents the frame index, c represents the camera index, and H×W represent the width and height of the frame, respectively. Each frame I f,c All have 3D skeleton poses annotated Where D represents the degrees of freedom of the 3D skeleton pose. From the skeleton pose θ f Extracting motion representation A sliding window is used from frame f-k+1 to frame f, and the heel joint translation is normalized to frame f. An implicit surface reconstruction method is used to recover the true ground geometry V from each frame. gt,f And render it to the depth map of all camera views.

[0047] When training animable Gaussian textures, the complex "one-to-many" mapping between skeletal motion and surface geometry—meaning similar skeletal poses can lead to significantly different surface deformations—makes it difficult to accurately predict surface geometry using only skeletal motion. This results in inaccurate alignment of the Gaussian texture in the depth direction, a phenomenon known as depth misalignment. This misalignment hinders accurate supervision of the Gaussian texture because it may be incorrectly projected onto the training image, leading to inaccurate colors and details in the rendered result.

[0048] To address the depth misalignment problem, a learnable latent code z is introduced for each frame. f , to move bones and potential code z f Together, they serve as input to a coarse-model network (i.e., a neural network that generates coarse models) to regress the 3D vertices V of a specific character's coarse-model mesh. f It is used to model geometry at a rough level.

[0049] z f By introducing the latent code z f This method can effectively solve the depth misalignment problem. Using latent codes along with skeletal motion as input allows the coarse-model network to capture more complex motion-to-geometric mappings, thereby improving the accuracy of the coarse-model mesh.

[0050]

[0051] Among them, z f It is the learnable latent code for each frame. Used to capture random geometric variations that cannot be explained by skeletal motion. H lat It is a shallow multilayer perceptron (MLP) used to generate latent codes. It is a positional encoding function that combines the frame index f and the maximum number of training frames f. max The ratio is encoded so that the model can learn the relative position information of the frames.

[0052] During training, the latent code z f The model is learned to capture the randomness in geometric changes. By combining the latent code with skeletal motion, the model is able to predict the geometry more accurately for each frame, thus reducing depth misalignment issues. During the inference phase, the latent code is set to zero so that new motion can be used to drive the character.

[0053] The mathematical expression:

[0054]

[0055] in, This represents a driveable coarse-model network, with skeletal motion as input. The output is the deformed 3D vertex set V. f (Also known as coarse-model mesh; note the distinction between coarse-model network and coarse-model mesh. A network is a neural network, while a mesh is a 3D mesh.) f dq This refers to dual quaternion skinning, a skinning technique used to deform vertices from a normal pose to a target pose. It combines rotation and translation, enabling more natural skin deformation. eg Embedded deformation is a technique for handling large-scale deformations. It achieves efficient deformation calculations by deforming in a low-dimensional space and then mapping it back to a high-dimensional space. eg Output the deformed vertex positions. S is the skinning weight matrix, which defines the weight of each vertex to different bones and is used to calculate the final vertex position during the skinning process. It is a coarse mesh under the standard orientation. eg It is an embedded deformation. It is a graph convolutional neural network used to predict deformation parameters A. f (i.e., the adjacency matrix of the deformed graph) and T f (i.e., the translation vector of the deformed diagram). It is a vertex displacement neural network used to predict the additional displacement δ of each vertex. f This refines the deformation results. θ f It represents the skeletal pose of the current frame and is a 3D pose vector that represents the character's current pose.

[0056] The above formula describes how skeletal motion is achieved through embedded deformation and dual quaternion skinning techniques. Mapped to a driveable coarse mesh This process involves multiple neural networks and geometric transformation techniques to ultimately generate 3D vertex positions that are consistent with the skeletal pose.

[0057] Surface drift: During training, the correspondence between the coarse mesh and the real surface drifts over time. The main reason is that implicit surface reconstruction methods lack a temporal correspondence between the ground geometry and the actual surface. Even if the depth misalignment problem is solved, drift may still exist between the coarse mesh and the real surface due to the Chamfer distance loss. It can be minimized in the event of an incorrect correspondence. This drift causes the Gaussian texture to lose consistency in position and color across different frames, resulting in a blurry average color and overly smooth surface distortion in the rendered output.

[0058] By utilizing a 2D point tracker and depth map, 2D correspondences are elevated to 3D space, and coarse meshes are optimized through vertex alignment loss, thereby improving the accuracy and coherence of geometric details. The vertex-level alignment method can handle complex clothing dynamics and texture patterns, effectively solving the surface drift problem.

[0059] 2D point tracking: Used to establish correspondences between consecutive frames. Specifically, for a rendered image with a coarse mesh and a real image, the drift of each vertex in 2D image space is calculated. 2D point tracking function The input consists of an image pair (I, I') and 2D image coordinates p, and the output is the matching point p'.

[0060] 2D to 3D rendering: Compare the rendered image of the coarse mesh with the training image and calculate the drift of each vertex in the 2D image space.

[0061] Rendering coarse-scale networks: using rasterizer R rast The deformed coarse mesh V f Rendered to image space.

[0062] I rast,f,c =R rast (V f ,T0,C c );

[0063] Among them, I rast,f,c This is the rendered image; T0 is the static texture of the coarse-matrix network; C... c These are camera parameters, V f It is the set of vertices in the current frame.

[0064] Calculate the 2D correspondence: for each vertex V f,i Calculate the 2D correspondence between the rendered image and the real image.

[0065] p f,c,i=c(I rast,f,c ,I f,c ,proj(V f,i C c ));

[0066] Among them, V f,i p is the i-th vertex of the current frame. f,c,i It is V f,i The corresponding 2D point in the camera's c view. c These are the camera parameters. `proj(·)` is a function that projects 3D points onto a 2D image. rast,f,c This is the rendered image. f,c The training image represents the image data captured by the camera at frame f and view c. It is an image frame in the multi-view video input and is used for subsequent processing and analysis.

[0067] See Figure 1 Consistency assessment: elevating 2D correspondences to 3D space and using visual... Figure 1 Consistency assessment is used to determine the most reliable 3D correspondence points.

[0068] Depth map query: For each camera view c, query the depth map N using bilinear interpolation. f,c The depth value in the text.

[0069]

[0070] Among them, P f,c,i Vertex V f,i At the corresponding 3D point p in camera view c f,c,i It is V f,i The corresponding 2D point in the camera's c-view. It is the inverse transformation matrix of the camera view c. bi_interpolation is the bilinear interpolation function.

[0071] View score calculation

[0072]

[0073] in, It is the score of camera view c with respect to vertex i. c It is the origin of the camera corresponding to camera view c. Vertex V f,i The normal line.

[0074] Best view selection

[0075]

[0076] Among them, v f,c,i Vertex V f,iVisibility in view c (e.g., 0 or 1). It is the score of camera view c for vertex i.

[0077] Final 3D Corresponding Points

[0078]

[0079] in, This represents the 3D corresponding point of the i-th vertex in the f-th frame, determined after viewing... Figure 1 After consistency evaluation, the most reliable 3D location is selected from multiple camera views. This indicates that in frame f, the selected camera view c is optimized. opt,f,i The determined 3D position of the i-th vertex is calculated based on the 2D point tracking results and depth information. opt,f,i This represents the optimal camera view index selected for the i-th vertex in the f-th frame among all camera views, used to determine the final 3D corresponding point.

[0080] Texture element-level alignment: To further address the potential drift issue in Gaussian textures, which affects the capture of the highest frequency details, this method improves detail preservation through texture element (texel)-level alignment.

[0081] Gaussian texture update: Building upon vertex-level alignment, an animated Gaussian texture representing the dynamic human appearance in detail is further trained. This is achieved by minimizing the loss function. To optimize Gaussian textures, enabling them to capture finer visual details.

[0082] Establishing texture element-level correspondences: To supervise Gaussian positions, an additional texture element-level correspondence loss is introduced based on their correspondence with the real surface. This loss function supervises Gaussian positions based on their correspondence. Its derivation method is similar to the 3D lifting method in vertex-level alignment, except that centroid interpolation is added to adapt to the texture level.

[0083] Detailed steps for establishing texture element-level correspondences;

[0084] 2D Correspondence Calculation: For each Gaussian texture element, calculate its correspondence in the 2D image space. This involves querying the corresponding points of the 2D point tracker between the rendered image and the real image.

[0085] 3D Upscaling: Upscaling 2D correspondences to 3D space by querying a depth map to obtain 3D positions. Specifically, for each texture element, its position in the image is used to query the depth map, and then the 2D position is converted into a point in 3D space using camera parameters.

[0086] Optimization of correspondence: By comparing the distance between the predicted Gaussian positions and the boosted 3D corresponding points, texture element-level correspondence loss is calculated. This loss function helps Gaussian texture elements align more accurately to the real surface.

[0087] Loss function of coarse-model networks (Including vertex-level alignment loss and texture element-level correspondence loss);

[0088]

[0089] in, It is the loss function of the coarse-model network, V f These are the vertices of the deformed coarse mesh, V. gt,f It is the vertex of the real geometry on the ground. It is the Chamfe distance loss. It is the vertex alignment loss. It is the texture element-level correspondence loss. It is the space regularization loss. λ corr-vertex , λ corr-tex and λ spatial These are the weighting coefficients for each loss term.

[0090] Chamfe distance loss Compare the distance between the deformed coarse mesh and the actual ground geometry.

[0091]

[0092] Among them, V f These are the vertices of the deformed coarse mesh, V. gt,f It is the vertex of the real geometry on the ground.

[0093] Vertex-level alignment loss The vertex alignment loss is calculated by comparing the distance between the predicted vertex and its boosted 3D counterpart.

[0094]

[0095] Among them, V f,i It is the predicted vertex position. It represents the enhanced 3D corresponding point, and F is the number of frames.

[0096] Texture element-level corresponding loss

[0097]

[0098] Where, μ i It is the center position of the Gaussian texture element. These are the enhanced 3D corresponding points used to supervise Gaussian positioning. F is the number of frames. V tex,f It is the set of valid texture elements in frame f.

[0099] Space regularization loss It includes Laplacian loss and surface normal consistency loss, used to maintain mesh smoothness.

[0100]

[0101] in, It is the Laplace transform. It is the Laplace smoothing term. It is the loss of surface normal consistency.

[0102] Laplace transform

[0103]

[0104] Here, Lap is the vertex Laplacian operator, used to calculate the difference between the average position of a vertex and its neighboring vertices. For vertex v i Its Laplace coordinates are N i It is vertex v i The number of neighboring vertices, It is vertex v i The set of neighboring vertices. V f It is the set of vertices in the current frame. It is the set of vertices after posing and skinning.

[0105] Laplace smoothing term

[0106]

[0107] Where Lap is the vertex Laplacian operator, Lap(V f ) i It is vertex v i Laplace coordinates, V f It is the set of vertices in the current frame, and N is the number of vertices.

[0108] Surface normal consistency loss

[0109]

[0110] Where, N tri N is the number of faces in the grid. tri,i It is the number of adjacent faces of face i. It is the normal to face i. It is the normal to the j-th adjacent face of face i. "·" represents the dot product, used to calculate the similarity between two normals.

[0111] Without physical constraints, clothing may exhibit issues such as tearing (e.g., arms piercing sleeves, fingers inserting into fabric), dynamic distortion (e.g., fabric tearing or abnormal gathering), or unnatural draping during vigorous movement (e.g., stiff / overstretched fabric, lacking material properties), especially in joint bending areas. To address these problems, this method embeds an implicit cloth physics solver during the coarse-model mesh vertex prediction stage.

[0112]

[0113] Among them, z f It is the learnable latent code for each frame. Used to capture random geometric changes that cannot be explained by skeletal motion. This represents a driveable coarse mesh with vertex-level alignment and texture element alignment. This is the output set of 3D vertices after deformation and physical correction. P is a differentiable projective physics calculator. λ∈[0,1] are adaptive physical weights, calculated using a multilayer perceptron and an activation function.

[0114] λ=σ(MLP(||Δθ f ||2,T material ));

[0115] Where σ represents the Sigmoid activation function, MLP represents the multilayer perceptron, and θ f Representing the skeletal pose of the current frame, it is a 3D pose vector that represents the character's current pose. Δθ f Indicates the rate of change in skeletal posture. T material Indicates the type of fabric material.

[0116] Differentiable cloth physics model: Transforms physical constraints into an optimizable problem, and combines efficient numerical methods to achieve a balance between physical accuracy and computational efficiency in cloth simulation.

[0117] Energy-based cloth model: Hyperelastic strain energy is constructed using continuum mechanics.

[0118] E cloth =E stretch +E bend +E cut ;

[0119] Among them, E cloth It is the total elastic potential energy of the fabric system, characterizing the fabric's ability to resist deformation. E stretch It is stretching energy, E bend It is bending energy, Ecut It is surface shear energy.

[0120]

[0121] Among them, e ij k represents the mesh edge connecting vertices i and j, and is the basic structural unit of the cloth mesh. s It is the tensile stiffness coefficient, which represents the strength to resist changes in length (e.g., denim > silk). and represents the spatial positions of vertex i and the j-th vertex, indicating the real-time deformation state of the cloth mesh. ||·||2 is the Euclidean norm, used to calculate the actual distance between vertices. This represents the length of the side in its static state, which is the original length of the side before deformation.

[0122]

[0123] Where, Δ ijk The triangular facet formed by vertices i, j, and k represents the discretization unit of the cloth surface. b φ is the bending stiffness coefficient, representing the strength to resist changes in curvature (e.g., thicker fabrics > thinner yarns). ijk This represents a real-time dihedral angle, which is the angle between adjacent triangular faces. This represents a stationary dihedral angle, which is the angle between the faces when they are not deformed.

[0124]

[0125] Where, k a It is the shear stiffness coefficient, representing the strength to resist in-plane distortion. A ijk It represents the surface area of ​​the current triangle. U ijk It is a real-time deformation gradient used to describe in-plane deformation. This is the reference deformation gradient, representing the baseline state before deformation. ||·|| F It is the Frobenius norm.

[0126] Implicit Euler Solver: Implicit Euler solvers are equivalent to minimizing the following objective function:

[0127]

[0128] Among them, V n+1 The objective is to determine the vertex position at the next moment. Δt is the time interval step, and M is the mass matrix (a diagonal matrix, the mass of each vertex). E cloth It is the elastic potential energy of the fabric. It is a predicted position, that is, a preliminary position considering only inertia and external forces. Where V n It is the current vertex position. M is the current vertex velocity, and M is the mass matrix. It is an external force (such as gravity, wind, etc.).

[0129] The objective function is solved using differentiable calculus, and its gradient is calculated. Setting the gradient to zero yields the nonlinear equilibrium equation (since elastic forces are generally nonlinear). This equation can typically be solved using Newton's iteration method, which requires calculating the Hessian matrix. Since the Hessian matrix is ​​usually symmetric and positive definite, the conjugate gradient method can be used to solve for linear systems, ultimately yielding the objective function V. n+1 (Note: The solution process is not the core content of this patent; it is only briefly described in words here.)

[0130] Collision handling mechanism: The collision handling mechanism achieves a balance between physical realism and computational efficiency through accurate distance field calculation and adaptive energy function, solving the most challenging collision problem in digital human clothing simulation.

[0131] Skeletal distance field construction;

[0132]

[0133] in, It is a point The shortest distance to the skeletal system indicates how close a point in space is to the surface of the skeleton. Indicates the position of the fabric vertex. b is the skeletal index. It is a point The nearest projection vector to bone b.

[0134]

[0135] in, It is the distance field gradient, pointing in the direction of the fastest increase in distance (i.e., the direction away from the bone). b * It is the skeleton that achieves the minimum distance, that is On point On the bones, gradient Undefined, requires special handling.

[0136] Continuous collision response;

[0137]

[0138] in, It's the repulsive force generated by the collision that prevents the fabric from penetrating the bone. c It is the collision stiffness coefficient (e.g., k) c =10 N / m), representing the intensity of the repulsive force. d safe It is the safe distance threshold (e.g., d) safe =0.3cm), which is the distance at which the repulsive force begins to be generated. ξ is the softening coefficient (e.g., ξ = 0.5cm), used to control the rate of force decay (which can be regarded as the range of force action; the smaller ξ is, the smaller the torque). It is the vertex The shortest distance to the bone.

[0139] Self-collision handling;

[0140]

[0141] Among them, E self It is a self-collision energy term, used to punish the fabric for self-penetration. k r d is the repulsion coefficient, representing the strength of self-repulsion. min It is the minimum spacing (e.g., d) min =0.1cm), which represents the minimum allowable distance between fabric layers. and This represents the position of different fabric vertices, which are potential collision point pairs.

[0142]

[0143] Where, k ro It is the basic repulsion coefficient (e.g., k) ro =1N / m 2 ),A avg It is the average area of ​​adjacent faces. ∈ represents the minimum value to prevent division by zero (e.g., ∈ = 10). -5 m).

[0144] Material Adaptive System: By more realistically simulating the physical behavior of various cloth materials under different motion states, the realism of the rendering is significantly improved.

[0145] Physical parameter prediction network: Based on the UV coordinates of the vertices and the fabric type, predict the parameters of the fabric in the physical simulation (tensile stiffness, bending stiffness, shear stiffness).

[0146]

[0147] Where, k s It is the tensile stiffness coefficient, representing the ability of a fabric to resist tensile deformation. b It is the bending stiffness coefficient, representing the ability of a fabric to resist bending deformation. aIt is the shear stiffness coefficient, representing the fabric's ability to resist angular deformation. MLP stands for Multilayer Perceptron for Material Parameters, consisting of three fully connected neural network layers (layer 1: 128 neurons + ReLU activation function + layer 2: 64 neurons + ReLU activation function + layer normalization + fully connected layer + Softplus activation function). UV i e is the texture coordinate of vertex i, indicating the vertex's position on the cloth surface. type It is a fabric type embedding vector, a 32-dimensional learnable vector.

[0148] Dynamic damping model: The damping coefficient is dynamically adjusted based on the UV coordinates of the vertex and the current velocity. Damping is used to simulate the energy dissipation of fabric during motion (such as air resistance and internal friction).

[0149]

[0150] Among them, c d It is the dynamic damping coefficient, representing the rate of energy dissipation controlling the movement of the fabric. base It is the basic damping coefficient (e.g., c) base =0.1N·s / m), representing the minimum damping value at rest. τ is the velocity sensitivity coefficient (e.g., τ = 0.05), representing the strength of the effect of controlling the motion velocity on the damping. It is the norm of the rate of change of skeletal posture. MLP is a damped prediction multilayer perceptron (2-layer fully connected network, input UV coordinates, output damping adjustment factor).

[0151] When the character moves rapidly (e.g., running), the speed of posture change is high, and the damping coefficient increases, simulating the additional energy dissipation of the fabric during rapid movement (e.g., increased air resistance). When the character is stationary or moving slowly, the damping coefficient is close to the baseline value, simulating only basic internal friction. Different areas of the fabric can have different damping (e.g., the edge of a sleeve may have greater damping than the torso), which is controlled by UV coordinates, as UV coordinates can be used to locate different parts of the fabric.

[0152] Real-time acceleration technology: Employs Hessian pre-computation, GPU parallelism, and LOD strategy. Specifically, when the skeletal motion change is less than 5°, the matrix of the previous frame is reused and GPU parallelism is used; when the distance to the camera is greater than 5 meters, it switches to a low-precision mesh (reducing the number of patches by 8 times).

[0153] Loss function: The loss function in this section will be discussed in the "Total Loss Function" section later. and

[0154] Gaussian texture;

[0155] Animable Gaussian texture parameters: In the UV space of the coarse-matrix network, each triangle-covered pixel stores a 3D Gaussian sputtering parameter (3DGS) used to capture the fine appearance and geometric details of clothing (such as texture patterns and wrinkles). Each 3DGS contains the following parameters:

[0156] Location From the deformed coarse mesh using barycentric interpolation Obtained from.

[0157] Rotation q uv,i : Indicates the rotation of Gaussian sputtering.

[0158] Anisotropic scaling uv,i : Indicates the scaling of Gaussian sputtering.

[0159] Opacity α uv,i : Indicates the opacity of Gaussian sputtering.

[0160] spherical harmonic coefficient η uv,i : Represents the color information of Gaussian sputtering.

[0161] Learnable motion-aware offset To model fine details (such as wrinkles) of dynamic clothing, a learnable motion-aware offset is applied to each Gaussian sputter in the normal space.

[0162] Predicting Gaussian texture parameters: via two independent convolutional neural networks ε app (Appearance decoder) and ε gen (Geometric decoder) predicts motion-aware Gaussian texture parameters.

[0163]

[0164] ε gen (T n ,T p )=(η uv,i );

[0165] Among them, T n It is the normal texture of a coarse mesh, T p It's a positional texture of a coarse-scale mesh. (Appearance decoder ε) app (T n ,T p Regression prediction of Gaussian sputtering position offset Rotation q uv,i Scaling uv,i and opacity α uv,i Geometric encoder ε gen (T n ,Tp Regression prediction of the spherical harmonic coefficient η of Gaussian sputtering uv,i .

[0166] Motion-aware offset: Motion-aware offset of Gaussian textures These are static quantities predicted independently for each frame, failing to model the dynamic evolution of clothing wrinkles, such as wrinkle propagation (e.g., sleeve folds spreading from the elbow to the wrist) and wrinkle merging / splitting (interactions between multiple folds). This leads to flickering or discontinuity in dynamic details during rapid motion. To address this issue, this method introduces a lightweight spatiotemporal Transformer module to establish cross-frame correlations in UV space, enabling dynamic perception of offsets.

[0167]

[0168] in, It is the motion feature matrix of the t-th frame. N n D is the number of nodes in the deformed graph. m It is the dimension of motion characteristics. G tex,t G is the Gaussian texture feature tensor of frame t. tex,t =MLP(q) uv,t ,s uv,t ,α uv,t ), where q uv,t It is the rotation quaternion (UV space) in the Gaussian parameters, s uv,t It refers to the anisotropic scaling and α in the Gaussian parameters. uv,t It is the opacity in the Gaussian parameters. It is the predicted residual offset, superimposed on the base offset. Transformer is a spatiotemporal Transformer encoder (3 layers, 4 heads of attention). Concat is the operation of concatenating two matrices.

[0169] Spatiotemporal attention mechanism;

[0170] Locally constrained attention: The attention range is limited to a local neighborhood with a UV space radius (r = 5 pixels), which conforms to the local propagation characteristics of wrinkle evolution.

[0171]

[0172] Among them, Q i It is a query vector, derived from the feature vector x of the current texture element i. i Through linear transformation matrix W Q Obtained by projection. K j It is a key vector, formed by the eigenvector x of the neighborhood element j. j Through linear transformation matrix W K V is obtained through projection.j It is a value vector, formed by the eigenvectors x of the neighborhood element j. j Through linear transformation matrix W V Obtained by projection. This represents the local spatial neighborhood of UV (i.e., the set of texture elements that satisfy spatial and temporal constraints). UV i These are the position coordinates of texture element i in UV space, uv j It represents the position coordinates of texture element j in UV space, ||uv j -uv i ||2 represents the Euclidean distance between the current texture element i and the candidate element j in the UV plane, where r is the spatial radius (Euclidean distance threshold, controlling the spatial neighborhood range; default r = 5 pixels). t i It is the sequence number (time index) to which texture element i belongs, t j `j` represents the sequence number (temporal index) of texture element `j`, and `Δt` represents the temporal radius (temporal difference threshold, controlling the temporal neighborhood range; default Δt = 1 frame). `d` represents the feature scaling factor, equal to the dimension of the key vector. For silk / thin fabrics, due to the fine and rapidly changing wrinkles, `r = 3` and `Δt = 1` are recommended; for thick coats / wool, due to the large and long-lasting wrinkles, `r = 8` and `Δt = 2` are recommended; for fast-moving clothing, to avoid motion blur, only the current frame is needed, so `r = 4` and `Δt = 0` are recommended; for slow, fine-grained animation of clothing, the range can be expanded to capture subtle changes, so `r = 6` and `Δt = 3` are recommended.

[0173] Multi-head attention decomposition;

[0174] MultiHead(Q,K,V)=Concat(head1,head2,…,head h W O ;

[0175]

[0176] Where m∈[1,h] represents the m-th attention head, and there are a total of h attention heads (e.g., h=4). Concat concatenates the outputs of multiple attention heads. W O This represents a linear transformation, where each attention head undergoes an independent linear transformation. and Calculate its attention score. Dimensions of each attention head. (e.g. d) m =64), d model It is the model dimension.

[0177] Feature fusion and residual mechanism;

[0178]

[0179] in, This represents the fused feature output of the l-th layer, where LayerNorm is the layer normalization. It represents the fused features from the previous layer (via residual connections), and CrossAttn is a cross-attention mechanism. Q is the query matrix. It's an appearance feature input; KV stands for key-value pairs. It is a motion feature input.

[0180] ε app and ε gen loss function

[0181]

[0182] Among them, I f,c It's rendering an image. It is a real image. It is L1 loss. It is a loss of time consistency. It is structural similarity loss. It is perceived loss.

[0183] L1 loss Compare the pixel differences between the rendered image and the real image.

[0184]

[0185] Among them, I f,c It's rendering an image. H and W are the height and width of the real image (or the rendered image, both of which are the same size). x and y are the pixel indices in the width and height directions of the image.

[0186] Time consistency loss;

[0187]

[0188] in, Denotes a differentiable warp operator based on optical flow, φ f Ω represents the optical flow field from frame f-1 to frame f, and Ω represents the effective texture region mask.

[0189] Structural similarity loss Evaluate the structural similarity between rendered images and real images.

[0190]

[0191] Perceived loss Features extracted from pre-trained deep networks are used to evaluate perceptual differences in images.

[0192]

[0193] Where φ is the pre-trained feature extraction network. f,c It's rendering an image. H and W are the height and width of the real image (or the rendered image, both of which are the same size). x and y are the pixel indices in the width and height directions of the image.

[0194] Total loss function;

[0195]

[0196] in, It is the rendering loss (i.e.) The loss function for the "Gaussian texture" part, It is the physical regularization loss (the loss function for the "physical cloth" part). This is the boundary constraint loss (the loss function for the "physical cloth" part). λ1 and λ2 represent weighting coefficients that control the constraint strength.

[0197] Physical regularization loss

[0198]

[0199] in, It is a balance constraint term. It's a collision penalty item. It is a motion consistency term.

[0200] Balance constraints

[0201]

[0202] in, It is the gradient operator with respect to the vertex position V, E cloth It is the fabric deformation energy (including tensile / bending / shear energy). To bring the system to a state of static equilibrium, that is...

[0203] Collision Penalty

[0204]

[0205] in, It is the vertex Distance field to the skeleton, d safe It is the safe distance threshold. This causes a linear penalty to be applied when a vertex penetrates a bone.

[0206] Motion Consistency Item

[0207]

[0208] in, It is the vertex acceleration, M is the diagonal mass matrix, F total Total external force (i.e., gravity + wind resistance + collision force), β is the motion weighting coefficient (e.g., β = 0.1). Enforce Newton's second law.

[0209] Boundary constraint loss

[0210]

[0211] Among them, v j These are the vertices on the fabric that need to be fixed. It refers to the corresponding anchor point location on the body. Boundary losses include constraints on clothing anchor points (such as collars fixed to the neck) and frictional constraints at the boundaries of clothing contact with the body.

[0212] Progressive training strategy: In order to stabilize the training process and avoid physical constraints interfering with network convergence in the early stage of training, a three-stage training approach is adopted.

[0213] Phase 1 (1-10k iterations): Freeze the physics algorithm (i.e., do not compute physics regularization terms), and train only the coarse-model network (including the learnable parameters of the coarse-model network and the Gaussian texture parameters). The goal is to allow the coarse-model network to initially learn the motion and basic geometry of the human body, ignoring physical details.

[0214]

[0215] Phase Two (10-50k iterations): Unfreeze the physical solver, but only enable balance constraints. and motion consistency Advanced collision handling is not enabled (because the geometry is not accurate enough at this point, and collision handling may introduce noise). The goal is to optimize the geometry under basic physical constraints, making the movement of clothing conform to the laws of physics, but tolerating some penetration.

[0216]

[0217] Phase 3 (greater than 50k iterations): Enable advanced collision handling and boundary loss. At this point, the physical regularization term... All terms (including collision penalties) are used in training, and the boundary loss is used. Also enabled. The goal is to fine-tune and resolve penetration issues, while ensuring that boundary conditions (such as clothing anchor points) are met.

[0218]

[0219] Analytical splatting is employed to render Gaussian textures to image space. This technique converts 3D Gaussian sputtering into a 2D image by integrating the contribution of each sputtering element to the actual pixel size. Analytical splatting provides better anti-aliasing than traditional tile-based rasterization, which is particularly important for supervising super-resolution data and preserving detail. During rendering, the parameters of each Gaussian sputtering element (position, rotation, scaling, opacity, and color) are used to calculate its projection onto the image plane and combined with camera parameters to generate the final rendered image. This process ensures that the Gaussian texture maintains sharpness and detail during rendering, without significant quality degradation even when viewed at higher resolutions.

[0220] R analytical (T gs C c ) = I R ;

[0221] Among them, T gs It is a Gaussian texture parameter, C c These are camera parameters, I R It is the rendered image.

[0222] On the other hand, this embodiment also provides an electronic device, including a memory, a processor, and a computing program stored in the memory and executable on the processor, wherein the processor implements the method when executing the computing program.

[0223] On the other hand, this embodiment also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method.

[0224] The above are merely preferred embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for generating high-precision dynamic digital humans, characterized in that, include: Each frame of the video sequence and its corresponding 3D skeleton pose are obtained from the multi-view video sequence. Frame-level skeletal motion features are extracted based on the three-dimensional skeletal pose. A learnable latent code is generated based on the skeletal motion features and frame index; The latent code and skeletal motion features are input into the coarse model network to obtain the three-dimensional vertices of the coarse model mesh for each frame. Based on the coarse model mesh, geometric deformation is performed by combining double quaternion skinning and embedded deformation methods to obtain the deformed coarse model mesh. Based on the deformed coarse mesh, image rendering is performed and 2D point tracking is performed in conjunction with the real image to obtain the correspondence between vertices in the image space. The optimal corresponding point of each vertex in the three-dimensional space is obtained based on the 2D correspondence and the depth map; Based on the corresponding 3D points, a vertex-level alignment loss function is constructed to optimize the position of the vertices in the coarse mesh. Based on the optimized vertex positions, construct the texture element-level correspondence and establish a loss function to obtain an animated Gaussian texture; Based on the Gaussian texture, it is rendered into the image space by the analytical Gaussian sputtering method to generate a high-resolution dynamic digital human image; It also includes: inputting the optimized coarse mesh vertices into the cloth physics calculation module, constructing the energy function of the cloth system based on the physical parameters generated by the skeleton pose, material type and latent code, and solving the vertex positions of the cloth after deformation based on the energy function, which is used to correct the three-dimensional geometry of dynamic clothing; The energy function includes tensile energy, bending energy and shear energy. The solution process is based on the implicit Euler method and uses the conjugate gradient method to minimize the objective function containing the mass matrix, elastic potential energy and predicted inertia term to obtain the equilibrium vertex displacement that satisfies the physical constraints. It also includes: based on the obtained initial Gaussian texture parameters, further constructing the spatiotemporal residual offset based on inter-frame motion features and appearance features, and using the spatiotemporal Transformer module to perform local attention modeling in the UV space to obtain the texture element offset adjustment amount between consecutive frames; The spatiotemporal Transformer module constructs neighborhood constraints using UV radius and temporal frame difference, and fuses motion features with Gaussian texture features through a multi-head attention mechanism to output residual offsets used to correct the temporal consistency of texture elements.

2. The method according to claim 1, characterized in that, The process of extracting frame-level skeletal motion features based on the three-dimensional skeletal pose includes: normalizing the joint translation of the three-dimensional skeletal pose, and performing motion modeling on multiple frames of images using a sliding window.

3. The method according to claim 1, characterized in that, The process of generating a learnable latent code based on the skeletal motion features and frame index includes: the latent code is encoded by the frame index and skeletal motion features through a positional encoding function, and then input into a shallow multilayer perceptron to generate a model of geometric random changes that cannot be explained by skeletal motion.

4. The method according to claim 1, characterized in that, The coarse model network outputs a set of three-dimensional coarse model vertices in the standard pose based on the input skeletal motion features and latent codes. The coarse model network includes a graph convolutional embedding deformation module and a vertex translation neural network.

5. The method according to claim 4, characterized in that, The dual quaternion skinning technique is used to deform a mesh in a standard pose into a mesh in a target pose. The embedded deformation is predicted by a graph convolutional neural network to determine the vertex translation vector and the adjacency structure.

6. The method according to claim 1, characterized in that, The rendered image and the real image are compared using a two-dimensional point tracker to establish the drift of vertices in the image space, and the three-dimensional points are projected into two-dimensional image points using camera parameters. The optimal 3D corresponding point is obtained by bilinear interpolation of the depth map, and the most reliable 3D point is selected from multiple camera views as the final corresponding point in combination with the view consistency score. The vertex-level alignment loss includes the predicted Euclidean distance between the vertex and the optimal corresponding 3D point, and the loss function is used in conjunction with the Chamfe distance for mesh optimization.

7. An electronic device comprising a memory, a processor, and a computing program stored in the memory and executable on the processor, characterized in that, When the processor executes the computing program, it implements the method of any one of claims 1-6.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1-6.

Images