Digital human reconstruction method with high-fidelity triangular mesh and material texture map

Abstract

The application discloses a digital human reconstruction method with high-fidelity triangular mesh and material texture mapping, and belongs to the technical field of computer graphics and digital human reconstruction. S1: performing space point sampling on each frame of picture corresponding to monocular video based on ray tracing, and deforming the sampling points to distribution under a standard posture; S2: acquiring geometric information and color information of global space points; S3: performing integration on the sampling points on each light ray to obtain volume rendering results, and completing first-stage optimization; S4: selecting a target frame, initializing a three-dimensional mesh, and generating a human body geometric surface; S5: acquiring material texture properties of the corrected human body geometric surface through a material network; S6: realizing differentiable rendering on the corrected human body geometric surface; S7: introducing an information fusion strategy to generate dense body rendering results under a virtual perspective, and supervising second-stage optimization; and S8: finally generating a digital human with a high-quality triangular mesh surface and material texture properties.

Description

Technical Field

[0001] This invention belongs to the field of computer graphics and digital human reconstruction technology, specifically relating to a digital human reconstruction method with high-fidelity triangular meshes and material texture maps. Background Technology

[0002] Digital human avatars are widely used in various scenarios such as virtual reality and film and television production. A digital human avatar that meets standard industrial applications includes multiple attributes, including geometry, human texture parameters, and material decomposition mapping. Therefore, producing or modeling a high-fidelity digital human typically requires expensive photographic equipment and a complex technical pipeline.

[0003] In recent years, relying on the concept of neural radiation fields, neural implicit representation, as a novel neural network-based reconstruction representation method, has achieved superior reconstruction results. Its main feature is based on volume rendering equations, where the density and color information of light sampling points in space are represented and trained using two fully connected networks. During inference, light rays are emitted and sampled from a known camera viewpoint. The density and color fields are used to query the information of the sampling points for volume rendering integration, thereby reconstructing the rendered image from that viewpoint. This optimization strategy does not rely on 3D information supervision; it only requires calculating the loss based on the 2D rendered image and ground truth, and then backpropagating the gradient to the network to optimize the network parameters, thus fitting a density and color field that conforms to the human body. This not only ensures reconstruction quality while eliminating the expensive initial data acquisition process but also greatly simplifies the reconstruction pipeline.

[0004] Based on this technology, various methods for implicitly reconstructing digital humans have been proposed. However, these methods generally suffer from the following problems: they rely on multiple camera perspectives; the implicit results have a narrow range of applications and are not compatible with traditional rendering engines, only outputting rendered results; the reconstructed results have rough geometric surfaces that do not conform to the real human body surface; the reconstructed surface color information lacks physical properties, making it impossible to render the desired color in different environments; and the reconstructed results do not have two-dimensional texture information, making extended editing and applications complex and difficult.

[0005] All of the above places higher demands on high-fidelity digital human reconstruction, namely, the reconstruction must be carried out within a limited data cost, and the reconstruction must be compatible with standard graphics engines and have physically attributed texture maps. Summary of the Invention

[0006] To reconstruct a digital human with physically-defined texture maps compatible with standard graphics engines under only two-dimensional information supervision, this invention proposes a digital human reconstruction method with high-fidelity triangular meshes and texture maps. Addressing the lack of multi-view supervision in explicit geometry and texture reconstruction, this method proposes an information fusion strategy that integrates image information from multiple frames in a video sequence into spatial supervision information. This invention can generate digital humans with high-quality triangular mesh surfaces and texture attributes, and is compatible with standard graphics engines.

[0007] The technical solution adopted in this invention is as follows:

[0008] A method for reconstructing a digital human with high-fidelity triangular meshes and material texture maps includes the following steps:

[0009] S1: Perform ray-traced spatial point sampling on each frame of the monocular video, and use a deformation field to deform the human body surface sampling points under different poses in each frame to the distribution under the standard pose, so as to obtain the global spatial points aggregated under the standard pose.

[0010] S2: Based on the global spatial points aggregated to the standard pose, the geometric and color information of the global spatial points under the standard pose are represented by the neural SDF field and the color field, respectively.

[0011] S3: Integrate the sampling points on each ray to obtain the volume rendering result, calculate the loss between the rendering result and the true value of the input frame, and continuously optimize the deformation field, neural SDF field and color field by forward propagating the gradient.

[0012] S4: Select the target frame from the monocular video and initialize the 3D mesh. Obtain the SDF value corresponding to each vertex of the initialized 3D mesh by querying the neural SDF field. Extract the isosurface with an SDF value of zero to represent the geometric surface of the human body.

[0013] S5: Introduce learnable human geometry surface deviation to correct the triangular mesh of the human geometry surface, and obtain the material texture properties of the corrected human geometry surface through the material network.

[0014] S6: Based on the lighting equation and material texture properties, differentiable rendering is achieved on the corrected human body geometry surface;

[0015] S7: Introduce an information fusion strategy to generate dense virtual volume rendering results, calculate the loss between differentiable rendering results and dense virtual volume rendering results, and jointly optimize learnable human geometric surface deviations and material networks.

[0016] S8: Based on the optimized deformation field, neural SDF field, color field, human geometric surface deviation, and material network, generate a digital human with high-quality triangular mesh surface and material texture properties.

[0017] Furthermore, in step S1, the sampling points on the human body surface under different poses in each frame are deformed to the distribution under the standard pose using a deformation field. The calculation process is expressed as follows:

[0018] x′=T i (x)+D i (T i (x), p(i))

[0019] Among them, T i (.), D i (.) represent the rigid and non-rigid deformations corresponding to the sampling points of the i-th frame image, respectively, x represents the sampling point, p(i) represents the smpl pose parameters corresponding to the i-th frame image, and x′ represents the point in the standard pose space.

[0020] Furthermore, the calculation process for the rigid deformation is expressed as follows:

[0021]

[0022]

[0023] Δw=f Δw (x, ψ) i )

[0024] Where K represents the number of joints in the human body. and This represents the rotation and displacement of the j-th joint of the human body in the i-th frame image. f represents the skin weight corresponding to the j-th joint of the human body in the i-th frame image. Δw (.) represents the neural-skin residual network, Δw represents the skin residual value, and ψ i It is the latent variable corresponding to the i-th frame in the neural skin residual network. It is the initial skin weight of the j-th joint of the human body in the i-th frame image, obtained by standard smpl prior surface nearest neighbor interpolation.

[0025] Furthermore, the calculation process of the neural SDF field is expressed as follows:

[0026] s(x′), z(x′)=f s (x′)

[0027] Among them, f s (.) represents the neural SDF field, x′ represents a point in standard pose space, and s(.) and z(.) represent the SDF value and geometric information, respectively.

[0028] Furthermore, the calculation process of the color field is expressed as follows:

[0029]

[0030] Among them, f c (.) represents the color field, x′ represents a point in standard pose space, and n(.), z(.), and v(.) represent the normal vector, geometric eigenvector, and ray direction of point x′ in standard pose space, respectively. c represents the latent variable corresponding to the i-th frame of the image. i (.) indicates color information.

[0031] Furthermore, the volume rendering result is obtained by integrating the sampling points on each ray, and the calculation process is expressed as follows:

[0032]

[0033] α n =1-exp(-σ n δ n )

[0034] α m =1-exp(-σ m δ m )

[0035] in, This represents the volume rendering result, where N represents the number of sampling points on each ray, and c i，n α represents the color of the nth sample point on each ray of the i-th frame image. n α m Describing the density distribution, δ n σ represents the sampling interval of the nth sampling point. n This represents the density of the nth sampling point.

[0036] Furthermore, the density of the sampling points is calculated based on the SDF value obtained from the neural SDF field, and the calculation process is expressed as follows:

[0037]

[0038] Where σ(.) represents the density of sampling points, α and β > 0 are learnable parameters, s(.) represents the sdf value, and x′ represents a point in the standard attitude space.

[0039] Furthermore, the calculation process of step S5 is expressed as follows:

[0040] k d r, m = f t (x′ s)

[0041] x′ s =x s -f0*n s

[0042] Among them, f t (.) represents the material mesh, x′ s Let x represent the vertex of the optimized human body geometry. s Let f0 represent the vertices of the human body geometry surface before optimization, and f0 represent the learnable deviation of the human body geometry surface. s k represents the direction of the normal at the vertex of the geometric surface of the human body. d r and m represent the base color, surface roughness, and surface metallicity, respectively.

[0043] Furthermore, the information fusion strategy introduced to obtain a dense virtual volume rendering result specifically includes:

[0044] Based on the viewpoint parameters of the target frame selected from monocular video, a dense virtual volume rendering result C is generated under multiple virtual viewpoints, utilizing optimized deformation field, neural SDF field, and color field. Virtual ={C V1 C V2 C VM}, C VM This indicates the number of times the Mth virtual object is rendered.

[0045] Furthermore, the deformation field, neural SDF field, color field, and material network are constructed using an MLP network.

[0046] The beneficial effects of this invention are mainly reflected in:

[0047] (1) This invention proposes a new end-to-end pipeline that can reconstruct a digital human avatar with high-fidelity three-dimensional geometric surfaces and material texture maps with only two-dimensional image information. Its explicit results are compatible with various standard graphics engines for further application and editing.

[0048] (2) This invention proposes an information fusion strategy that integrates the image information of multiple frames in the video time sequence into spatial supervision information, thereby alleviating the viewpoint sparsity problem encountered in optimization under explicit field.

[0049] (3) This invention extracts geometric surfaces through implicit fields and further optimizes surface texture material information based on rasterized explicit differentiable rendering, which not only accelerates the training process, but also effectively reconstructs more detailed material texture information that conforms to physical properties. Attached Figure Description

[0050] Figure 1This is a flowchart of the digital human reconstruction method with high-fidelity triangular mesh and material texture mapping proposed in this invention;

[0051] Figure 2 This invention relates to the reconstruction results and applications under real data input.

[0052] Figure 3 This is a schematic diagram illustrating the implementation framework of the digital human reconstruction method with high-fidelity triangular mesh and material texture mapping proposed in this invention. Detailed Implementation

[0053] The present invention will be further described below with reference to the accompanying drawings and some examples.

[0054] This invention proposes a digital human reconstruction method with high-fidelity triangular meshes and material texture maps, the flowchart of which is shown below. Figure 1 As shown, the entire process is divided into two stages. The first stage integrates multi-frame temporal image information into global color information in a standard space by training a variable implicit neural human representation, while also supporting the extraction of volumetric surfaces after training convergence. The second stage, based on differentiable rendering, jointly optimizes geometry and explicit surface material textures under the supervision of an information fusion strategy. Figure 2 The reconstruction results are shown by inputting a set of real human videos, demonstrating that the reconstruction results of the present invention have high fidelity and are compatible with standard graphics engines.

[0055] like Figure 3 As shown, the specific implementation steps are as follows:

[0056] Step 1: Sample the pixels of the input frame image along the light direction to obtain sampling points x. Pass the densely packed sampling points x in space through a deformation field to obtain a point x′ in the standard pose space, where the deformation field includes both rigid and non-rigid deformations. The deformation field is defined as follows:

[0057] x′=T i (x)+D i (T i (x), p(i))

[0058] Among them, T i (x) represents the rigid deformation corresponding to sampling point x in the i-th frame image, D i (T i (x), p(i)) represent the non-rigid deformation of the spatial point corresponding to the i-th frame image. The input of the non-rigid deformation network includes the rigid deformation and the pose parameter p(i) of the smpl corresponding to the i-th frame. In this embodiment, the non-rigid deformation process is implemented using a multilayer perceptron (MLP) network.

[0059] The T i (x) is implemented as follows:

[0060]

[0061] Where K represents the number of joints in the human body. and This represents the rotation and displacement of the j-th joint of the human body in the i-th frame image. This represents the skinning weight corresponding to the j-th joint in the i-th frame image. Skinning weight The definition is as follows:

[0062]

[0063] Where, Δw=f Δw :(x, ψ i ) represents the neural skin residual network f Δw The output skin residual value, ψ i It is the latent variable corresponding to the i-th frame in the neural skin residual network. The initial skinning weights for the j-th joint in the i-th frame image are obtained based on the standard smpl prior surface nearest neighbor interpolation. In this embodiment, the neural skinning residual network f Δw Implemented by an MLP network.

[0064] Step 2: Set up a neural symbolic distance field (SDF field) to represent the geometry of the human body surface, and optimize the color information in space through a color field.

[0065] The neural SDF field is defined as follows:

[0066] s(x′), z(x′)=f s (x′)

[0067] Among them, the neural field f s The input (.) is a point x′ in the deformed standard pose space, and the output is the sdf value s(x′) and the feature vector z(x′) representing geometric information.

[0068] The color field is defined as follows:

[0069]

[0070] The input to the color field includes a point x′ in the deformed standard pose space, the normal vector n(x′) of x′, the geometric feature vector z(x′) of x′, the ray direction v(x′) of x′, and the latent variable corresponding to the i-th frame. Output the color information c corresponding to the sampling point x i .

[0071] In this embodiment, the neural SDF field and color field mentioned above are both implemented by an MLP network.

[0072] Step 3: Based on the volume rendering equation, integrate the sampling points on each ray to obtain the rendering value of the pixel on the ray under the viewpoint. The volume rendering equation of the pixel on the ray is defined as follows:

[0073]

[0074] in, This represents the volume rendering result, where N represents the number of sampling points on each ray, and c i，n α represents the color of the nth sample point on each ray of the i-th frame image. n =1-exp(-σ n δ n ) represents the density distribution, δ n =t n+1 -t n Represents the sampling interval, σ n This represents the density of the nth sampling point.

[0075] The density value of the sampling point is obtained by transforming the SDF value using a transformation formula. Taking sampling point x as an example, the transformation is performed based on the SDF value of its corresponding point x′ in the standard pose space, as shown in the following formula:

[0076]

[0077] Here, α and β > 0 are both learnable parameters.

[0078] Rendering results The gradient is backpropagated to the neural skin residual network, non-rigid deformation, neural SDF field, and color field layers to continuously optimize the corresponding geometry and color.

[0079] Step 4: After the implicit neural expression network converges, select a suitable pose frame with minimal occlusion as the reconstruction target frame. Create a 256-bit network. 3 A 3D mesh with a resolution of [resolution value missing] is used, and the SDF value of each vertex is queried through a neural SDF field. Based on the SDF values, the potential human body surface can be represented as an isosurface with an SDF value of zero, defined as follows:

[0080] S = {x: f(x) = 0}

[0081] Therefore, high-fidelity triangular surface meshes can be extracted based on marching cubes.

[0082] Step 5: Use a material network to represent the physical material information of the human body's geometric surface. In this embodiment, the material network is implemented using an MLP network, defined as follows:

[0083] k d r, m = f t (x′ s )

[0084] Among them, material network f t The input to (.) is the vertex x′ of the optimized human body geometry. s The output includes the basic color k. d The surface roughness r and surface metallicity m, along with the base color and metallicity, further affect the color information of the reflected light. The color of the reflected light is defined as follows:

[0085] k s = (1-m)*0.04+m*k d

[0086] The texture mapping relationship based on the triangular mesh surface can unfold the material information in the material network output into a two-dimensional texture map, and further perform differentiable rendering based on the texture map.

[0087] While optimizing material textures, the entire pipeline simultaneously optimizes the geometric surface. This is achieved by introducing a learnable bias, f. o This is used to offset the deviation of the SDF field near the zero isosurface. This deviation is only applied in the vertex normal direction of the geometric surface, thus not altering the characteristics of the geometry itself. The optimization formula is defined as follows:

[0088] x′ s =x s -f0*n s

[0089] Where, n s Vertex x representing the geometric surface of the human body s The direction of the normal at that location.

[0090] Step 6: Implement explicit rasterized differentiable rendering based on the PBR lighting equation. The lighting equation is defined as follows:

[0091] L o (x′ s ω o )=∫ Ω fr(x′ s ω i ω o )L i (x′ s ω i )(ω i ·n(x′s ))dω i

[0092] Where, ω i It is the incident direction, ω o It is the direction of emission, x′ s It is the optimized geometric surface vertex, f r (x′ s ω i ω o ) is a BRDF item, L i (x′ s ω i ) represents the radiation along the incident direction, and the integration domain Ω is the surface normal n(x′) around the intersection point. s (Hemisphere). To accelerate the integration process, an approximate representation of illumination using decomposition and summation is adopted, and the formula is simplified as follows:

[0093] L o (x′ s ω o )=∫ Ω f r (x′ s ω i ω o )(ω i ·n(x′ s ))dω i *∫ Ω f r (x′ s ω i ω o )D i (x′ s ω i )(ω i ·n(x′ s ))dω i

[0094] The first integral term represents the integration of the specular BRDF with the fixed ambient light, and the second integral term represents the integration of the incident radiation with the specular NDF. Both of these terms can be pre-integrated, greatly reducing the computational cost of the network's forward propagation. A learnable hexahedron is used to represent the ambient illumination, and the incident illumination is obtained by indexing the intersection of the incident direction of the geometric surface with the surface pixels of the hexahedron.

[0095] Step 7: Supervise the rendering results of explicit differentiable rendering under multiple viewpoints using an information fusion strategy to address the viewpoint sparsity problem under explicit human geometry surface optimization. Based on the viewpoint parameters of the selected reconstructed target frame, generate dense virtual volume rendering results C under multiple virtual viewpoints using the neural implicit neural representation field trained in the first stage. Virtual ={C V1C V2 C V50}; Calculate the differentiable rendering result C with dense virtual volume rendering results Virtual The loss is calculated by backpropagating the gradient and jointly optimizing the learnable human geometric surface bias and material network.

[0096] Step 8: Based on the optimized deformation field, neural SDF field, color field, human geometric surface deviation, and material network, generate a digital human with high-quality triangular mesh surface and material texture properties.

[0097] like Figure 2 and Figure 3 In one specific embodiment of the present invention, the input video data is a monocular human dynamic rotation image taken in a real scene, with a total of 300 frames of images used as training input.

[0098] Depend on Figure 3 It is evident that this invention can effectively integrate dynamic human body information from each frame of a monocular camera into a standard spatial pose within a dynamic implicit neural representation field, and render and recreate realistic virtual results from multiple perspectives in a specified pose space as a supplementary fusion dataset. The neural SDF field of this invention can extract high-fidelity smooth triangular mesh surfaces. The explicit representation optimization of this invention learns realistic texture maps and geometric surfaces that conform to real-world images. Figure 2 It is understood that the present invention can optimize and generate high-fidelity human geometry and material texture maps under the supervision of a single frame of video with only two-dimensional image information, and the generated explicit results can be directly compatible with standard graphics engines, enabling more applications and editing, including texture editing, new action generation, and new perspective generation.

[0099] The embodiments described herein are merely illustrative examples of implementations of the inventive concept. The scope of protection of this invention should not be considered limited to the specific forms described in these examples, but also extends to equivalent technical means conceived by those skilled in the art based on the inventive concept.

Claims

1. A method for reconstructing a digital human with high-fidelity triangular meshes and material texture maps, characterized in that, Includes the following steps: S1: Perform ray-traced spatial point sampling on each frame of the monocular video, and use a deformation field to deform the human body surface sampling points under different poses in each frame to the distribution under the standard pose, so as to obtain the global spatial points aggregated under the standard pose. S2: Based on the global spatial points aggregated to the standard pose, the geometric and color information of the global spatial points under the standard pose are represented by the neural SDF field and the color field, respectively. S3: Integrate the sampling points on each ray to obtain the volume rendering result, calculate the loss between the rendering result and the true value of the input frame, and continuously optimize the deformation field, neural SDF field and color field by forward propagating the gradient. S4: Select the target frame from the monocular video and initialize the 3D mesh. Obtain the SDF value corresponding to each vertex of the initialized 3D mesh by querying the neural SDF field. Extract the isosurface with an SDF value of zero to represent the geometric surface of the human body. S5: Introduce learnable human geometry surface deviation to correct the triangular mesh of the human geometry surface, and obtain the material texture properties of the corrected human geometry surface through the material network. The calculation process of step S5 is expressed as follows: ； ； in, Represents the material network. Represents the vertices of the optimized human body geometry. This represents the vertices of the human body's geometric surface before optimization. This represents the learnable deviation of the human body's geometric surface. This indicates the direction of the normal at the vertex of the geometric surface of the human body. These represent the base color, surface roughness, and surface metallicity, respectively. S6: Based on the lighting equation and material texture properties, differentiable rendering is achieved on the corrected human body geometry surface; S7: Introduce an information fusion strategy to generate dense virtual volume rendering results, calculate the loss between differentiable rendering results and dense virtual volume rendering results, and jointly optimize learnable human geometric surface deviations and material networks. The information fusion strategy introduced to generate dense virtual volume rendering results specifically includes: Based on the viewpoint parameters of the target frame selected from monocular video, dense virtual volume rendering results under multiple virtual viewpoints are generated using optimized deformation fields, neural SDF fields, and color fields. , This indicates the number of times the Mth virtual object is rendered; S8: Based on the optimized deformation field, neural SDF field, color field, human geometric surface deviation, and material network, generate a digital human with high-quality triangular mesh surface and material texture properties.

2. The digital human reconstruction method with high-fidelity triangular mesh and material texture mapping according to claim 1, characterized in that, In step S1, the human surface sampling points under different poses in each frame are deformed to the distribution under the standard pose using a deformation field. The calculation process is as follows: ； in, Let represent the rigid deformation and non-rigid deformation corresponding to the sampling points of the i-th frame image, respectively. Indicates the sampling point. This represents the pose parameters of smpl corresponding to the i-th frame image. This represents a point in standard pose space.

3. The digital human reconstruction method with high-fidelity triangular mesh and material texture mapping according to claim 2, characterized in that, The calculation process for the rigid deformation is expressed as follows: ； ； ； Where K represents the number of joints in the human body. and This represents the rotation and displacement of the j-th joint of the human body in the i-th frame image. This represents the skin weight corresponding to the j-th joint of the human body in the i-th frame image. Represents the neural skin residual network. Indicates the skin residual value. It is the latent variable corresponding to the i-th frame in the neural skin residual network. It is the initial skin weight of the j-th joint of the human body in the i-th frame image, obtained by standard smpl prior surface nearest neighbor interpolation.

4. The digital human reconstruction method with high-fidelity triangular mesh and material texture mapping according to claim 1, characterized in that, The calculation process of the neural SDF field is expressed as follows: ； in, Represents the neural SDF field. Represents a point in standard pose space. These represent the SDF value and geometric information, respectively.

5. The digital human reconstruction method with high-fidelity triangular mesh and material texture mapping according to claim 4, characterized in that, The calculation process for the color field is expressed as follows: ； in, Represents color field, Represents a point in standard pose space. Representing points in standard pose space The normal vector, geometric eigenvectors, and ray direction, This represents the latent variable corresponding to the i-th frame of the image. Indicates color information.

6. The digital human reconstruction method with high-fidelity triangular mesh and material texture mapping according to claim 1, characterized in that, The volume rendering result is obtained by integrating the sampling points on each ray, and the calculation process is expressed as follows: ； ； ； in, This represents the volume rendering result, where N represents the number of sampling points on each ray. Let n represent the color of the nth sample point on each ray of the i-th frame of the image. Represents density distribution, This represents the sampling interval of the nth sampling point. This represents the density of the nth sampling point. This represents the sampling interval of the m-th sampling point. This represents the density of the m-th sampling point.

7. The digital human reconstruction method with high-fidelity triangular mesh and material texture mapping according to claim 6, characterized in that, The density of the sampling points is calculated based on the SDF value obtained from the neural SDF field. The calculation process is as follows: ； in, Indicates the density of sampling points. All of these are learnable parameters. This represents the SDF value. This represents a point in standard pose space.

8. The digital human reconstruction method with high-fidelity triangular mesh and material texture mapping according to claim 1, characterized in that, The deformation field, neural SDF field, color field, and material network mentioned above adopt MLP network.

Images