Human body grid reconstruction method, device and equipment and storage medium

Abstract

The invention discloses a human body grid reconstruction method and device, equipment and a storage medium, and relates to the technical field of computers. The method comprises the following steps: based on a single human body image, generating a multi-view human body image and a corresponding multi-view normal vector map by using a multi-view diffusion model; based on the multi-view human body image and the multi-view normal vector chartlet, generating a transition human body grid through differential rendering and continuous re-gridding; converting the transition human body grid into a target posture human body grid, performing patch completion based on the target posture human body grid, and migrating a linear hybrid skin weight to the target posture human body grid to obtain a drivable human body grid; and back-projecting the color of the multi-view human body image to the texture space of the drivable human body grid to obtain a texture grid. Refined drivable human body grid reconstruction can be realized, and texture recovery can be realized.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a method, apparatus, device, and storage medium for human body mesh reconstruction. Background Technology

[0002] Human body mesh refers to the geometry of the human body surface represented by a 3D triangular mesh; single-image-based human body mesh reconstruction refers to estimating the 3D mesh of the human body from a single 2D image. Actuable mesh refers to a reconstructed 3D human body mesh that can be manipulated to perform new movements by changing its pose parameters (such as joint rotation).

[0003] Among related technologies, human body mesh reconstruction algorithms based on single images include ECON, SiTH, and PSHuman; however, there are still problems such as unsatisfactory restoration of the real human body shape, lack of texture, or inability to achieve driving. Summary of the Invention

[0004] In view of this, the purpose of this invention is to provide a method, apparatus, device, and storage medium for human body mesh reconstruction, capable of achieving refined, driveable human body mesh reconstruction and texture restoration. The specific solution is as follows: In a first aspect, this application discloses a method for reconstructing a human body mesh, comprising: Based on a single human body image, a multi-view human body image and corresponding multi-view normal vector map are generated using a multi-view diffusion model. Based on the multi-view human body image and the multi-view normal vector map, a transitional human body mesh is generated through differentiable rendering and continuous remeshing. The transitional human body mesh is converted into a target pose human body mesh, patch completion is performed based on the target pose human body mesh, and the linear hybrid skinning weights are transferred to the target pose human body mesh to obtain a driveable human body mesh. The colors of the multi-view human body image are back-projected onto the texture space of the driveable human body mesh to obtain a texture mesh.

[0005] Optionally, the step of generating multi-view human images and corresponding multi-view normal vector maps using a multi-view diffusion model includes: Extract the face region image from the human body image; The face region image and the human body image are denoised using a multi-view diffusion model based on perspective information and at preset time steps. During the denoising process, based on the denoised human body image at step t and the denoised face region image at step t-1, the denoised human body image at step t-1 is predicted, until all time steps are completed to obtain multi-view human body images and multi-view normal vector maps.

[0006] Optionally, predicting the denoised human image at step t-1 based on the denoised human image at step t-1 and the denoised face region image at step t-1 includes: By encoding the face region image and the human body image, corresponding face latent variables and human body latent variables are obtained; After scaling the face latent variable after denoising in step t-1, add it to the face region of the human latent variable after denoising in step t to obtain the updated human latent variable after denoising in step t. The denoised human latent variables after the t-th step of the update are denoised to obtain the denoised human latent variables after the (t-1)-th step.

[0007] Optionally, the step of generating a transitional human body mesh based on the multi-view human body image and the multi-view normal vector map through differentiable rendering and continuous remeshing includes: An initial human body mesh is generated based on the multi-view human body images using SMPL-X estimation. Differentiable rendering is performed on the initial human body mesh to obtain a rendering normal vector map; Based on the loss between the rendered normal map and the multi-view normal map, the mesh vertices of the initial human body mesh are iteratively optimized, and re-meshed after each optimization to obtain the transitional human body mesh.

[0008] Optionally, converting the transitional human mesh into a target pose human mesh includes: Obtain an SMPL-X mesh with the same pose as the transitional human body mesh, traverse the vertices in the transitional human body mesh, and determine the first matching vertex corresponding to the current vertex in the SMPL-X mesh; Based on the first matching vertex, and combined with the transformation relationship between the SMPL-X mesh and the target pose SMPL-X mesh, the second matching vertex corresponding to the current vertex in the target pose SMPL-X mesh is determined, and the target pose human body mesh is obtained based on the second matching vertices corresponding to all vertices. The step of transferring linear hybrid skin weights to the target pose human mesh to obtain a driveable human mesh includes: The linear hybrid skinning weights of the target pose SMPL-X mesh are transferred to the target pose human body mesh to obtain the drivable human body mesh.

[0009] Optionally, the step of back-projecting the colors of the multi-view human body image onto the texture space of the driveable human body mesh includes: The driveable human body mesh is sequentially driven to the poses of each of the multi-view human body images, and the color of the current multi-view human body image is back-projected onto the texture space of the driveable human body mesh. Based on the projected texture space, texture completion is performed using Laplacian smooth diffusion.

[0010] Optionally, converting the transitional human mesh into a target pose human mesh includes: The topology of the human hand in the transitional human body mesh is replaced with the topology of the human hand in the target pose SMPL-X mesh. After back-projecting the colors of the multi-view human body image onto the texture space of the driveable human body mesh, the method further includes: The hand texture mesh template is migrated to the texture space of the driveable human body mesh.

[0011] Optionally, migrating the hand texture mesh template to the texture space of the driveable human body mesh includes: Based on the difference between the hand color of the hand texture mesh template and the hand color of the multi-view human body graphic, a color conversion function is constructed using the color conversion method; The hand texture mesh template is migrated to the texture space of the driveable human body mesh using the color conversion function.

[0012] Secondly, this application discloses a human body mesh reconstruction device, comprising: The multi-view generation module is used to generate multi-view human images and corresponding multi-view normal vector maps based on a single human image using a multi-view diffusion model. The rendering optimization module is used to generate a transitional human body mesh based on the multi-view human body image and the multi-view normal vector map through differentiable rendering and continuous remeshing. The weight transfer module is used to convert the transitional human body mesh into a target pose human body mesh, perform patch completion based on the target pose human body mesh, and transfer the linear hybrid skin weights to the target pose human body mesh to obtain a driveable human body mesh. The texture optimization module is used to backproject the colors of the multi-view human body image onto the texture space of the driveable human body mesh to obtain a texture mesh.

[0013] Thirdly, this application discloses an electronic device, including: Memory, used to store computer programs; A processor is used to execute the computer program to implement the aforementioned human body mesh reconstruction method.

[0014] Fourthly, this application discloses a computer-readable storage medium for storing a computer program; wherein the computer program, when executed by a processor, implements the aforementioned human body mesh reconstruction method.

[0015] In this application, a multi-view human image and a corresponding multi-view normal vector map are generated based on a single human image using a multi-view diffusion model; a transitional human mesh is generated based on the multi-view human image and the multi-view normal vector map through differentiable rendering and continuous re-meshing; the transitional human mesh is converted into a target pose human mesh, patch completion is performed based on the target pose human mesh, and the linear blending skin weights are transferred to the target pose human mesh to obtain a driveable human mesh; the colors of the multi-view human image are back-projected onto the texture space of the driveable human mesh to obtain a textured mesh.

[0016] As shown above, after generating multi-view human images and corresponding multi-view normal vector maps through a multi-view diffusion model, the human mesh is updated through differentiable rendering and continuous re-meshing. The accuracy and comprehensiveness of the completion are improved by converting the transitional human mesh into the target pose human mesh and then performing patch completion. Furthermore, by transferring the linear blending skin weights to the target pose human mesh, the mesh and skeleton are bound together, resulting in a drivable human mesh. Finally, the texture mesh is obtained by back-projecting the colors of the multi-view human images onto the texture space of the drivable human mesh. This system supports fine-grained human mesh reconstruction from input human images of arbitrary poses, automatically binding the mesh to the skeletal structure, and also supports texture reconstruction of the human mesh, thus obtaining a complete 3D human model with drivable characteristics. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0018] Figure 1 A flowchart of a human body mesh reconstruction method provided in this application; Figure 2 This application provides a specific flowchart for generating multi-view human body images; Figure 3 A schematic diagram illustrating a specific multi-view human body image generation result provided in this application; Figure 4 A schematic diagram illustrating a specific multi-view normal vector map generation result provided in this application; Figure 5This application provides a specific flowchart for constructing a transitional human body mesh; Figure 6 This application provides a specific initial human body mesh diagram; Figure 7 This application provides a specific schematic diagram of a transitional human body mesh; Figure 8 This application provides a specific flowchart for driving human body mesh generation; Figure 9 This application provides a specific target pose human body mesh diagram; Figure 10 This application provides a specific diagram illustrating the mesh completion result and its driving effect. Figure 11 A flowchart of a specific texture optimization method provided in this application; Figure 12 This application provides a schematic diagram of a specific texture optimization result; Figure 13 A specific hand replacement flowchart is provided for this application; Figure 14 This application provides a schematic diagram of a specific hand replacement result; Figure 15 A flowchart of a specific human body mesh reconstruction method provided in this application; Figure 16 This application provides a schematic diagram of the structure of a human body mesh reconstruction device; Figure 17 This application provides a structural diagram of an electronic device. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] In existing technologies, ECON obtains a partial human body mesh through bilateral normal vector integration and then uses Poisson reconstruction to synthesize a watertight human body mesh. However, the distance between the front and rear human body meshes cannot be controlled during the synthesis process. Too close or too far will result in the reconstructed human body mesh being too flat or too bulky, making it difficult to restore the real human body shape. Furthermore, ECON only focuses on human body mesh reconstruction and does not restore the mesh texture. SiTH first generates a back image of the human body through a diffusion model, then obtains the normal vector maps corresponding to the front and rear human body images through a normal vector estimator. It projects the 3D points in space onto the pixel space to obtain the corresponding features and estimates the SDF (Signed Distance Function) and color through a neural network. It then uses Poisson reconstruction to convert the SDF into a human body mesh and obtains the vertex color through a multilayer perceptron. However, SiTH uses a multilayer perceptron to estimate the SDF of spatial points and uses Poisson reconstruction to obtain the human body mesh, resulting in a relatively smooth mesh that cannot completely correspond to the estimated human body normal vector map. At the same time, SiTH uses a multilayer perceptron to estimate the vertex color of the mesh. Although it can fill in the color of invisible areas, the resulting texture is relatively coarse, and the reconstruction results of details such as the face and clothing are not refined enough. Furthermore, SiTH outputs a static implicit mesh without bones or parametric structures, thus only providing a human body mesh corresponding to the image pose, and cannot drive the reconstructed mesh. PSHuman's reconstructed human body mesh also cannot be driven. Moreover, due to self-occlusion in the input human body image and issues such as consistent pitch angles in the generated multi-view images, the texture of some mesh vertices cannot be recovered, and the reconstruction results for the hands are not ideal. To overcome these technical problems, this application proposes a human body mesh reconstruction method that can achieve refined, driveable human body mesh reconstruction and supports high-fidelity texture reconstruction.

[0021] This application discloses a method for human body mesh reconstruction. See also Figure 1 As shown, the method may include the following steps: Step S11: Based on a single human body image, generate multi-view human body images and corresponding multi-view normal vector maps using a multi-view diffusion model.

[0022] A diffusion model is used to obtain human images and normal vector maps from different perspectives based on the input human image, which are then used for subsequent human mesh reconstruction and texture restoration. For example... Figure 2 As shown, the input human image is detected and cropped to obtain a face image, which, along with multiple copies of the human image, is input into a multi-view diffusion model. Multiple copies are used to generate images from multiple perspectives. For example, based on the input human image, six perspectives (front, left-front, left, back, right, right-front) are generated, each perspective being generated in parallel, conditioned on the input image. The result is as follows. Figure 3The multi-view human body images shown, and Figure 4 The multi-view normal vector mapping, the above multi-view diffusion model can be built based on the open-source pre-trained text image model SD2.1-unclip, and trained using multi-view images rendered from an open-source 3D human mesh dataset. Of course, any other feasible method is acceptable. This embodiment does not specifically limit the type of diffusion model.

[0023] The method of generating multi-view human images and corresponding multi-view normal vector maps using a multi-view diffusion model specifically includes: extracting the face region image from the human image; denoising the face region image and the human image using a multi-view diffusion model at preset time steps, combining viewpoint information; during the denoising process, predicting the denoised human image at step t-1 based on the denoised human image at step t-1 and the denoised face region image at step t-1, until all time steps are completed to obtain the multi-view human image and multi-view normal vector map. It is understandable that since the face occupies only a small, low-resolution area in an image, it cannot be accurately generated using a conventional diffusion model; and humans are very sensitive to subtle changes in the face, inaccurate face generation can lead to obvious distortion and identity changes. Therefore, this embodiment uses a multi-view diffusion model to generate higher resolution and more accurate faces. Specifically, in the stepwise denoising process of the multi-view diffusion model, the intermediate result of the face image is scaled and fused with the corresponding region in the human body image. The human body image after denoising in step t-1 is generated by referring to the human body image after denoising in step t and the face region image after denoising in step t-1.

[0024] In other words, the image data distribution is defined as the joint distribution of human body images and face images: ; in, and These represent images of the human body and images of the human face, respectively.

[0025] The forward noise addition process in the diffusion model can be represented as: The forward noise addition process is part of the training process of the diffusion model. Since the influence of faces on the noise addition process is relatively small, the face image can be ignored when adding noise to human images. ; The backward denoising process of the diffusion model can be expressed as: ; Among them, face denoising is performed during the backward denoising process. Similar to the conventional diffusion model, but with human body denoising... The face denoising result is used as an additional condition, i.e., the human image after denoising in step t-1. It refers to the human body image after denoising in step t. And the face region image after denoising in step t-1 Generated.

[0026] In some embodiments, predicting the denoised human image at step (t-1) based on the denoised human image at step t and the denoised face region image at step (t-1) includes: encoding the face region image and the human image to obtain corresponding face latent variables and human latent variables; scaling the denoised face latent variables at step (t-1) and adding them to the face region of the denoised human latent variables at step t to obtain updated denoised human latent variables at step t; and denoising the updated denoised human latent variables at step t to obtain denoised human latent variables at step (t-1). Joint noise is performed using a noise mixing layer, and the updated denoised human latent variables at step t are: ; in, This represents the latent vector of the human body at time t (time step). This represents the latent face vector at time t-1 (time step). It is a binary mask for the face region, and R represents the scaling operation.

[0027] Specifically, in each self-attention module of the diffusion model, the latent vector of the face branch is extracted and scaled, and then added to the face region of the latent vector of the human body branch corresponding to the current self-attention module. Based on the latent vectors of the human body branches of all self-attention modules, the updated human body latent vector is obtained.

[0028] Specifically, this is achieved at the latent variable level. The input image is first encoded using VAE to obtain latent variables, and then copied into six copies according to the number of viewpoints. These copies are then concatenated with the latent variables obtained from the face image encoding in the batch dimension. Gaussian noise is concatenated with the latent variables in the channel dimension before being input into the model. Additional timestamps and text are input for variable modulation and cross-attention operations. For human image denoising, intermediate results from the face image are used. The face image is scaled and fused into the latent variables of the human image to achieve a more reasonable and consistent face generation result.

[0029] Step S12: Based on the multi-view human body image and the multi-view normal vector map, a transitional human body mesh is generated through differentiable rendering and continuous remeshing.

[0030] In this embodiment, after obtaining multi-view human images and multi-view normal vector maps through a multi-view diffusion model, a transitional human mesh corresponding to the pose is obtained through differentiable rendering and continuous re-meshing. For example... Figure 5The transitional human body mesh construction process shown first estimates two-dimensional keypoints and SMPL-X parameters based on multi-view images, and then further optimizes the SMPL-X parameters based on the two-dimensional keypoints. The optimized SMPL-X parameters are used to generate an SMPL-X mesh, which is used as an initial mesh for subsequent optimization. The initial mesh is then used to generate a rendering normal map through differentiable rendering. The loss between the rendering normal map and the multi-view normal map is calculated. The mesh is optimized based on the loss while continuously re-meshing, and finally, a transitional human body mesh for the corresponding pose is obtained.

[0031] In some embodiments, the step of generating a transitional human mesh based on the multi-view human image and the multi-view normal map through differentiable rendering and continuous re-meshing specifically includes: generating an initial human mesh based on the multi-view human image using SMPL-X estimation; performing differentiable rendering on the initial human mesh to obtain a rendered normal map; iteratively optimizing the mesh vertices of the initial human mesh based on the loss between the rendered normal map and the multi-view normal map, and re-meshing after each optimization to obtain the transitional human mesh.

[0032] Understandably, the multi-view image first estimates 2D keypoints and SMPL-X parameters using OpenPose and SMPLer-X. Then, it optimizes the SMPL-X parameters based on the 2D keypoints, setting them as optimizable parameters. An SMPL-X mesh (initial human body mesh) and keypoints are generated based on these parameters. The keypoints are then projected onto the multi-view image, and loss calculations are performed using the estimated 2D keypoints to further optimize the SMPL-X parameters. The initial human body mesh is shown below. Figure 6 As shown.

[0033] In related technologies, methods for human body reconstruction using normal vector maps include mapping the normal vector map to an implicit surface using implicit functions and using bilateral normal vector integrals in ECON. Implicit function mapping usually leads to the loss of fine geometric details and requires a lot of time, while bilateral normal vector integrals also significantly reduce the overall geometric effect. This application uses an explicit triangular mesh representation and optimizes the triangular mesh using normal vector maps. Differentiable rendering is used to transfer the image loss gradient to the human body mesh to optimize vertex positions. However, directly using the differentiable rendering gradient to optimize the mesh may lead to problems such as face flipping and self-intersection. In addition, meshes with fixed topology are difficult to use with complex geometries. Therefore, continuous re-meshing is adopted to apply adaptive mesh reconstruction in each iteration of the optimization loop to solve these problems. By continuously and adaptively re-meshing the mesh, the triangular faces are kept within the optimal range, avoiding problems such as face distortion and flipping, and complex geometries can be generated. Given SMPL-X as the initial human body mesh, differentiable rendering technology is used to render the normal vector map of the initial human body mesh, and loss is calculated with the obtained multi-view normal vector map. The specific loss function is as follows: ; in, This indicates the rendering of the normal map. This represents the silhouette of the initial human body mesh. This represents a multi-view normal vector map. This represents a multi-view normal vector map and silhouette, where i represents the i-th viewpoint human image; Represents the normal vector of a mesh vertex. This represents the average normal vector of adjacent vertices. Loss gradient backpropagation optimizes the mesh vertices, followed by remeshing to redivide the vertices and faces. After optimization, Poisson reconstruction is used to complete invisible small regions, such as the chin, resulting in... Figure 7 The transitional human body mesh shown corresponds to the pose.

[0034] As can be seen, differentiable rendering enables a differentiable mapping between image space and 3D space, allowing for backpropagation updates of the mesh geometry based on rendering errors during optimization. Next, continuous remeshing technology is employed to perform topology optimization and geometric refinement on the initial human body mesh, ensuring that the reconstructed model maintains structural integrity while possessing high-precision surface detail representation. Finally, a high-quality 3D human body mesh model corresponding to the input pose is obtained.

[0035] Step S13: Convert the transitional human body mesh into a target pose human body mesh, perform patch completion based on the target pose human body mesh, and transfer the linear blended skin weights to the target pose human body mesh to obtain a driveable human body mesh.

[0036] The reconstructed transitional human mesh corresponding to the pose of the input image cannot be driven because it does not have linear blend skinning weights (LBS). Therefore, in this embodiment, by aligning SMPL-X with the reconstructed mesh and migrating the LBS weights of SMPL-X to the reconstructed human mesh, the reconstructed human mesh can be driven like SMPL-X. The SMPL-X (Skinned Multi-Person Linear model eXtended) model is used to estimate the body pose (body pose): 21 main joints (63-dimensional axis angles); hand pose: 15 joints for each hand, a total of 30 joints (90 dimensions); facial expression; body shape, controlling height, weight, etc.; global rotation and translation, that is, 6D camera / world coordinate transformation.

[0037] In a preferred embodiment, the conversion of the transitional human mesh into a target pose human mesh includes: obtaining an SMPL-X mesh with the same pose as the transitional human mesh, traversing the vertices in the transitional human mesh, and determining the first matching vertex corresponding to the current vertex in the SMPL-X mesh; based on the first matching vertex, combining the transformation relationship between the SMPL-X mesh and the target pose SMPL-X mesh, determining the second matching vertex corresponding to the current vertex in the target pose SMPL-X mesh, and obtaining the target pose human mesh based on the second matching vertices corresponding to all vertices; the migration of the linear blend skinning weights to the target pose human mesh to obtain a drivable human mesh includes: migrating the linear blend skinning weights of the target pose SMPL-X mesh to the target pose human mesh to obtain the drivable human mesh. The above-mentioned patch completion based on the target pose human mesh includes: deleting the abnormal patches in the target pose human mesh, performing patch completion processing, and performing Poisson reconstruction based on the mesh after patch completion. The above-mentioned target pose can specifically be the 'big' pose, that is, the human body resembles the Chinese character '大'.

[0038] As Figure 8 shown, first, an SMPL-X mesh corresponding to the human mesh pose and a 'big' pose SMPL-X mesh are pre-generated. For each vertex in the transitional human mesh, the nearest point on the SMPL-X mesh is found as the first matching vertex. Since the vertices on the SMPL-X mesh and the target pose SMPL-X mesh are in one-to-one correspondence, the vertices of the transitional human mesh are transferred to the 'big' pose to obtain the target pose human mesh ('big' pose human mesh) according to the position of the nearest point. And the missing part is completed using the 'big' pose SMPL-X mesh, and a complete human mesh is obtained using Poisson reconstruction. Then, the LBS weights of the 'big' pose SMPL-X mesh are migrated to the reconstructed mesh to obtain a drivable human mesh.

[0039] Understandably, due to the uncertainty of the input human pose, the body parts of the reconstructed human mesh may be stuck together, and directly performing skeleton binding on the reconstructed transitional human mesh will result in incorrect binding. Therefore, the reconstructed human mesh is first converted into a 'large' pose human mesh to ensure that the body parts can be correctly separated before skeleton binding is performed; the transfer to the 'large' pose requires first transferring to the standard pose, and then transferring to the 'large' pose. The specific process for determining the target pose human mesh is as follows: For each vertex v on the transitional human body mesh, find the nearest point on the SMPL-X mesh. ; ; The centroid coordinates of the nearest point on the patch are used to interpolate the transformation matrix of each vertex of the SMPL-X mesh based on the coordinates; ; Where i,j,k represent the nearest points The barycentric coordinates, a, b, c represent respectively The three vertices of the face are given, and T represents the transformation matrix. The reconstructed human body mesh can be transferred to a 'large' pose based on the transformation matrix; Where v represents a vertex in the transitional human body mesh, This indicates that vertex v is transformed into a vertex in the 'large' pose human mesh. This represents the inverse of the transformation matrix from the standard pose of the SMPL-X mesh to the pose of the transitional human mesh. This represents the transformation matrix from the standard pose of the SMPL-X mesh to the 'large' pose.

[0040] Furthermore, after transitioning to the 'large' pose, overstretched surfaces need to be removed. This is done by comparing the length changes of each edge before and after the transition to identify and delete abnormally stretched surfaces. (Reference) Figure 9 This demonstrates intermediate results of converting the transitional human mesh to a 'large' pose, as well as the target pose human mesh after removing abnormally stretched patches.

[0041] However, deleting abnormally stretched patches will cause the mesh to become incomplete. Therefore, the 'large' pose SMPL-X mesh is aligned with the target pose human mesh, the missing parts are filled in using the 'large' pose SMPL-X mesh, and Poisson reconstruction is used to obtain the complete 'large' pose human mesh. Then, the nearest point on the 'large' pose SMPL-X mesh is found, and the LBS weights of SMPL-X are transferred to the human mesh in the same way to obtain a driveable human mesh. ; in, This represents the LBS weight of vertex v, where i, j, k represent the nearest points. The centroid coordinates are given, where a, b, and c represent the three vertices of the patch containing the nearest point. The mesh reconstructed after LBS weight transfer can be driven by parameter control, similar to SMPL-X. Figure 10 The image shows the results of mesh completion reconstruction and driving.

[0042] As can be seen, for the input human body mesh, the nearest point is searched for on the SMPL-X mesh of the corresponding pose for each mesh vertex. Based on the spatial relationship of the nearest point, the vertex coordinates of the human body mesh are mapped to the 'large' pose. Then, distorted patches are removed, and the missing geometric parts are completed using the 'large' pose SMPL-X mesh. A topologically continuous and structurally complete 3D human body mesh model is generated through Poisson reconstruction. Finally, the predefined LBS weights in the 'large' pose SMPL-X mesh are transferred to the reconstructed human body mesh to achieve precise binding between the mesh and the skeleton, thereby obtaining a human body mesh model with drivable characteristics.

[0043] Step S14: Back-project the colors of the multi-view human body image onto the texture space of the driveable human body mesh to obtain a texture mesh.

[0044] In this embodiment, after obtaining the driveable human body mesh, the input single image and the generated multi-view human body images are used to optimize and complete the texture. In a preferred embodiment, the colors of the multi-view human body images are back-projected onto the texture space of the driveable human body mesh, including: sequentially driving the driveable human body mesh to the poses of each of the multi-view human body images, and back-projecting the colors of the current multi-view human body image onto the texture space of the driveable human body mesh; based on the projected texture space, texture completion is performed using Laplacian smooth diffusion. Figure 11 As shown, the human body mesh can be driven to the pose corresponding to the image according to the SMPL-X parameters. Then, the colors of the multi-view human body image are projected onto the mesh vertices to obtain a partial texture mesh. Then, the texture of the known vertices is diffused to the unknown vertices through Laplacian smooth diffusion to complete the texture and finally obtain the full texture mesh.

[0045] Specifically, the driveable human body mesh is first driven by the human body image pose to obtain the corresponding pose human body mesh, and the corresponding pose human body mesh is projected onto the multi-view human body image generated by the diffusion model. The effective facets for each viewpoint are selected according to the projection relationship and the cosine value of the angle between the facet normal vector and the viewing direction. ; in, The visible face is determined by the viewing direction and the face position, where N represents the face normal vector and V represents the viewing direction. After finding the valid faces, the vertices of the valid faces are projected onto the image of each viewpoint according to the camera's intrinsic and extrinsic parameters, and then colors are sampled from the image based on the projection positions. ; ; Where K represents the camera's intrinsic parameters, T represents the camera's extrinsic parameters, p represents the 2D position of the vertex in the image, F represents bilinear interpolation based on position, and c represents the sampled color. The sampled colors are obtained for all viewpoint images, and a weighted average is performed to obtain the final color. ; Indicates the final color of the vertex. This represents the vertex sample color of the i-th viewpoint. The corresponding weight is calculated from the cosine of the angle between the vertex normal vector and the view direction, and the preset weight for each viewpoint: ; This represents the cosine of the angle between the vertex normal vector and the line of sight. This indicates the preset weight for each viewpoint: front and back viewpoints are 1, left and right viewpoints are 0.8, and left-front and right-front viewpoints are 0.4.

[0046] Due to the human pose and viewpoint of the input image, some vertices cannot obtain their colors from multi-view images. This application proposes to complete the unknown vertices using Laplacian smooth diffusion based on the known vertex colors. Laplacian smooth diffusion is an iterative method that gradually diffuses the known vertex colors to the unknown vertices using the Laplacian matrix of the mesh. ; in, This represents the matrix consisting of rows containing unknown vertices in the grid Laplacian matrix. 'o' represents the vertex count, initially initialized to 1 for known vertices and 0 for unknown vertices, and updated as the matrix expands. For example... Figure 12 The image shows a comparison of the before and after texture optimization and completion.

[0047] As can be seen, by utilizing the color information of multi-view human images, the image textures from each viewpoint are projected onto the mesh vertices through projection mapping, thereby obtaining an initial human mesh containing partial texture information. Furthermore, to address the issue of missing textures at occluded vertices, a Laplacian smooth diffusion algorithm is employed to propagate the color distribution of the mesh vertices, smoothly diffusing the color information of known texture regions to unknown regions, achieving spatial completion and continuous transition of textures.

[0048] As can be seen from the above, in this embodiment, after generating multi-view human images and corresponding multi-view normal vector maps through a multi-view diffusion model, the human mesh is updated through differentiable rendering and continuous re-meshing; by converting the transitional human mesh into the target pose human mesh, and then performing patch completion, the accuracy and comprehensiveness of the completion are improved; and by transferring the linear blending skin weights to the target pose human mesh, the mesh and skeleton are bound together, thus obtaining a drivable human mesh; finally, the colors of the multi-view human images are back-projected onto the texture space of the drivable human mesh to obtain a textured mesh. This method supports fine-grained human mesh reconstruction after inputting human images in any pose, automatically binding the mesh to the skeletal structure, and also supports texture reconstruction of the human mesh, thereby obtaining a complete 3D human model with drivable characteristics.

[0049] Based on the above embodiments, and considering the poor performance of hand and human body mesh reconstruction in related technologies, this application also proposes a reconstruction scheme for human hands, including hand structure replacement and texture transfer.

[0050] In a preferred embodiment, in step S13, converting the transitional human body mesh into a target pose human body mesh includes: replacing the hand topology of the human body in the transitional human body mesh with the hand topology of the human body in the target pose SMPL-X mesh; correspondingly, in step S14, after back-projecting the color of the multi-view human body image onto the texture space of the driveable human body mesh, the method further includes: migrating the hand texture mesh template to the texture space of the driveable human body mesh.

[0051] Because the hand area in the image was small, the reconstructed hand was not detailed enough, and the fingers could not be clearly separated. Therefore, a MANO template hand was used to replace the hand area. See also Figure 13 As shown, hand region replacement mainly consists of two parts: geometric replacement and texture replacement. When constructing a 'large' pose human body mesh, the hand is reconstructed using hand geometry replacement in SMPL-X to obtain a complete and drivable hand topology. During texture optimization and completion, the hand texture is replaced using the MANO hand texture mesh template to obtain the final mesh. After removing distorted faces, when completing missing faces using the 'large' pose SMPL-X mesh, the vertices of the hand in the reconstructed human body mesh can be deleted according to the correspondence, and the complete and drivable hand from the SMPL-X mesh can be connected for subsequent Poisson reconstruction.

[0052] ; in, This indicates the hand vertex to be deleted. This represents the point on the SMPL-X mesh that is closest to vertex v. This represents the vertices of the hand in the SMPL-X mesh. After replacement and Poisson reconstruction, the driveable human mesh contains the complete driveable hand topology.

[0053] Furthermore, the texture of the MANO hand texture mesh is transferred to the final mesh through hand texture replacement; ; in, This indicates the color of the hand mesh in the MANO template. This represents the hand color obtained from multi-view images, where T represents the color conversion function. See also Figure 14 The image shown is a comparison diagram before and after the template was replaced.

[0054] Because different people have different skin colors, the template's hand mesh only has a single color, therefore it is necessary to... Switch to Color space. In a preferred embodiment, the migration of the hand texture mesh template to the texture space of the driveable human body mesh includes: constructing a color conversion function based on the difference between the hand color of the hand texture mesh template and the hand color of the multi-view human body graphic using a color conversion method; and migrating the hand texture mesh template to the texture space of the driveable human body mesh using the color conversion function.

[0055] For example, color conversion can be achieved using the LAB color conversion method; ; in, This indicates a conversion from the RGB color space to the LAB color space. This indicates the conversion from LAB color space to RGB color space, where mean represents the mean and std represents the variance.

[0056] As can be seen, by replacing the hand region in the reconstructed human body mesh with template hand geometry and texture, a complete and drivable hand topology and high-quality hand texture are obtained. Furthermore, through color space conversion, the template hand texture is transformed according to the hand color in the input image to ensure the reasonableness of the hand texture.

[0057] See Figure 15The diagram illustrates a specific flowchart of a human body mesh reconstruction method. The process is as follows: First, a single input human body image is processed using a multi-view diffusion model to generate multi-view human body images and multi-view normal vector maps. After obtaining the multi-view normal vector maps, differentiable rendering and continuous remeshing are used to obtain a transitional human body mesh. The transitional human body mesh is aligned with SMPL-X, and the hand topology is replaced. The LBS weights of SMPL-X are then transferred to the reconstructed human body mesh to obtain a drivable human body mesh. Based on the generated multi-view human body images, the drivable human body mesh undergoes texture optimization. The colors of invisible vertices are filled in using Laplacian smooth diffusion, and texture transfer is performed based on the template hand to obtain the final textured mesh. This method solves the problems of undrivable reconstructed human body meshes, the inability to recover textures from some mesh vertices due to self-occlusion, and poor hand reconstruction results.

[0058] Accordingly, this application also discloses a human body mesh reconstruction device, see [link to relevant documentation]. Figure 16 As shown, the device includes: The multi-view generation module 11 is used to generate multi-view human images and corresponding multi-view normal vector maps based on a single human image using a multi-view diffusion model. Rendering optimization module 12 is used to generate a transitional human body mesh based on the multi-view human body image and the multi-view normal vector map through differentiable rendering and continuous remeshing. Weight transfer module 13 is used to convert the transitional human body mesh into a target pose human body mesh, perform patch completion based on the target pose human body mesh, and transfer the linear hybrid skin weights to the target pose human body mesh to obtain a driveable human body mesh. The texture optimization module 14 is used to backproject the colors of the multi-view human body image onto the texture space of the driveable human body mesh to obtain a texture mesh.

[0059] As can be seen from the above, in this embodiment, after generating multi-view human images and corresponding multi-view normal vector maps through a multi-view diffusion model, the human mesh is updated through differentiable rendering and continuous re-meshing; by converting the transitional human mesh into the target pose human mesh, and then performing patch completion, the accuracy and comprehensiveness of the completion are improved; and by transferring the linear blending skin weights to the target pose human mesh, the mesh and skeleton are bound together, thus obtaining a drivable human mesh; finally, the colors of the multi-view human images are back-projected onto the texture space of the drivable human mesh to obtain a textured mesh. This method supports fine-grained human mesh reconstruction after inputting human images in any pose, automatically binding the mesh to the skeletal structure, and also supports texture reconstruction of the human mesh, thereby obtaining a complete 3D human model with drivable characteristics.

[0060] In some specific embodiments, the multi-view generation module 11 may specifically include: The extraction unit is used to extract the face region image from the human body image; The denoising unit is used to combine viewpoint information and use a multi-view diffusion model to denoise the face region image and the human body image according to a preset time step. During the denoising process, based on the human body image after denoising at step t and the face region image after denoising at step t-1, the human body image after denoising at step t-1 is predicted until all time steps are completed to obtain the multi-view human body image and the multi-view normal vector map.

[0061] In some specific embodiments, the noise reduction unit may specifically include: The encoding unit is used to obtain corresponding face latent variables and body latent variables by encoding the face region image and the body image; The scaling addition unit is used to scale and add the latent face variables after the denoising step t-1 to the face region of the latent human body variables after the denoising step t, so as to obtain the updated latent human body variables after the denoising step t. The denoising unit is used to denoise the updated human latent variables after denoising at step t to obtain the human latent variables after denoising at step t-1.

[0062] In some specific embodiments, the rendering optimization module 12 may specifically include: An initial human body mesh generation unit is used to generate an initial human body mesh based on the multi-view human body image through SMPL-X estimation. Differentiable rendering unit, used to perform differentiable rendering on the initial human body mesh to obtain a rendering normal vector map; A remeshing unit is used to iteratively optimize the mesh vertices of the initial human body mesh based on the loss between the rendered normal map and the multi-view normal map, and to remesh after each optimization to obtain the transitional human body mesh.

[0063] In some specific embodiments, the weight transfer module 13 may specifically include: The first matching vertex determination unit is used to obtain an SMPL-X mesh with the same pose as the transitional human body mesh, traverse the vertices in the transitional human body mesh, and determine the first matching vertex corresponding to the current vertex in the SMPL-X mesh. The second matching vertex determination unit is used to determine the second matching vertex corresponding to the current vertex in the target pose SMPL-X mesh based on the first matching vertex and in combination with the transformation relationship between the SMPL-X mesh and the target pose SMPL-X mesh, and to obtain the target pose human body mesh based on the second matching vertices corresponding to all vertices. Accordingly, the weight transfer module 13 may specifically include: The weight transfer unit is used to transfer the linear hybrid skin weights of the target pose SMPL-X mesh to the target pose human body mesh to obtain the drivable human body mesh.

[0064] In some specific embodiments, the texture optimization module 14 may specifically include: The color back-projection unit is used to sequentially drive the driveable human body mesh to the poses of each of the multi-view human body images, and back-project the color of the current multi-view human body image to the texture space of the driveable human body mesh. The smooth diffusion unit is used for texture completion based on the projected texture space using Laplacian smooth diffusion.

[0065] In some specific embodiments, the weight transfer module 13 may specifically include: A hand replacement unit is used to replace the hand topology of the human body in the transitional human body mesh with the hand topology of the human body in the target pose SMPL-X mesh. Accordingly, the texture optimization module 14 may further include: A hand texture transfer unit is used to transfer a hand texture mesh template to the texture space of the driveable human body mesh after back-projecting the colors of the multi-view human body image onto the texture space of the driveable human body mesh.

[0066] In some specific embodiments, the hand texture transfer unit can be used to construct a color conversion function based on the difference between the hand color of the hand texture mesh template and the hand color of the multi-view human body graphic, using a color conversion method; and through the color conversion function, the hand texture mesh template is transferred to the texture space of the driveable human body mesh.

[0067] Furthermore, this application also discloses an electronic device, see [link to relevant documentation]. Figure 17 As shown, the content in the figure should not be considered as any limitation on the scope of use of this application.

[0068] Figure 17 This is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of this application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the human body mesh reconstruction method disclosed in any of the foregoing embodiments.

[0069] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0070] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored thereon include operating system 221, computer program 222 and data 223 including multi-view human images, etc. The storage method can be temporary storage or permanent storage.

[0071] The operating system 221 manages and controls the various hardware devices on the electronic device 20 and the computer program 222 to enable the processor 21 to perform calculations and processing on the massive data 223 in the memory 22. It can be Windows Server, Netware, Unix, Linux, etc. In addition to including computer programs capable of performing the human body mesh reconstruction method executed by the electronic device 20 as disclosed in any of the foregoing embodiments, the computer program 222 may further include computer programs capable of performing other specific tasks.

[0072] Furthermore, this application also discloses a computer storage medium storing computer-executable instructions. When the computer-executable instructions are loaded and executed by a processor, they implement the steps of the human body mesh reconstruction method disclosed in any of the foregoing embodiments.

[0073] Furthermore, this application also discloses a computer program product, including a computer program that, when executed by a processor, implements the human body mesh reconstruction method steps disclosed in any of the foregoing embodiments.

[0074] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0075] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0076] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0077] The present invention provides a detailed description of a human body mesh reconstruction method, apparatus, device, and storage medium. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, those skilled in the art will recognize that there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for reconstructing a human body mesh, characterized in that, include: Based on a single human body image, a multi-view human body image and corresponding multi-view normal vector map are generated using a multi-view diffusion model. Based on the multi-view human body image and the multi-view normal vector map, a transitional human body mesh is generated through differentiable rendering and continuous remeshing. The transitional human body mesh is converted into a target pose human body mesh, patch completion is performed based on the target pose human body mesh, and the linear hybrid skinning weights are transferred to the target pose human body mesh to obtain a driveable human body mesh. The colors of the multi-view human body image are back-projected onto the texture space of the driveable human body mesh to obtain a texture mesh.

2. The human body mesh reconstruction method according to claim 1, characterized in that, The generation of multi-view human images and corresponding multi-view normal vector maps using a multi-view diffusion model includes: Extract the face region image from the human body image; The face region image and the human body image are denoised using a multi-view diffusion model based on perspective information and at preset time steps. During the denoising process, based on the denoised human body image at step t and the denoised face region image at step t-1, the denoised human body image at step t-1 is predicted, until all time steps are completed to obtain multi-view human body images and multi-view normal vector maps.

3. The human body mesh reconstruction method according to claim 2, characterized in that, The step of predicting the denoised human image in step (t-1) based on the denoised human image in step t and the denoised face region image in step (t-1) includes: By encoding the face region image and the human body image, corresponding face latent variables and human body latent variables are obtained; After scaling the face latent variable after denoising in step t-1, add it to the face region of the human latent variable after denoising in step t to obtain the updated human latent variable after denoising in step t. The denoised human latent variables after the t-th step of the update are denoised to obtain the denoised human latent variables after the (t-1)-th step.

4. The human body mesh reconstruction method according to claim 1, characterized in that, The process of generating a transitional human body mesh based on the multi-view human body image and the multi-view normal vector map through differentiable rendering and continuous re-meshing includes: An initial human body mesh is generated based on the multi-view human body images using SMPL-X estimation. Differentiable rendering is performed on the initial human body mesh to obtain a rendering normal vector map; Based on the loss between the rendered normal map and the multi-view normal map, the mesh vertices of the initial human body mesh are iteratively optimized, and re-meshed after each optimization to obtain the transitional human body mesh.

5. The human body mesh reconstruction method according to claim 1, characterized in that, The step of converting the transitional human mesh into a target pose human mesh includes: Obtain an SMPL-X mesh with the same pose as the transitional human body mesh, traverse the vertices in the transitional human body mesh, and determine the first matching vertex corresponding to the current vertex in the SMPL-X mesh; Based on the first matching vertex, and combined with the transformation relationship between the SMPL-X mesh and the target pose SMPL-X mesh, the second matching vertex corresponding to the current vertex in the target pose SMPL-X mesh is determined, and the target pose human body mesh is obtained based on the second matching vertices corresponding to all vertices. The step of transferring linear hybrid skin weights to the target pose human mesh to obtain a driveable human mesh includes: The linear hybrid skinning weights of the target pose SMPL-X mesh are transferred to the target pose human body mesh to obtain the drivable human body mesh.

6. The human body mesh reconstruction method according to claim 1, characterized in that, The step of back-projecting the colors of the multi-view human body image onto the texture space of the driveable human body mesh includes: The driveable human body mesh is sequentially driven to the poses of each of the multi-view human body images, and the color of the current multi-view human body image is back-projected onto the texture space of the driveable human body mesh. Based on the projected texture space, texture completion is performed using Laplacian smooth diffusion.

7. The human body mesh reconstruction method according to any one of claims 1 to 6, characterized in that, The step of converting the transitional human mesh into a target pose human mesh includes: The topology of the human hand in the transitional human body mesh is replaced with the topology of the human hand in the target pose SMPL-X mesh. After back-projecting the colors of the multi-view human body image onto the texture space of the driveable human body mesh, the method further includes: The hand texture mesh template is migrated to the texture space of the driveable human body mesh.

8. The human body mesh reconstruction method according to claim 7, characterized in that, The migration of the hand texture mesh template to the texture space of the driveable human body mesh includes: Based on the difference between the hand color of the hand texture mesh template and the hand color of the multi-view human body graphic, a color conversion function is constructed using the color conversion method; The hand texture mesh template is migrated to the texture space of the driveable human body mesh using the color conversion function.

9. A human body mesh reconstruction device, characterized in that, include: The multi-view generation module is used to generate multi-view human images and corresponding multi-view normal vector maps based on a single human image using a multi-view diffusion model. The rendering optimization module is used to generate a transitional human body mesh based on the multi-view human body image and the multi-view normal vector map through differentiable rendering and continuous remeshing. The weight transfer module is used to convert the transitional human body mesh into a target pose human body mesh, perform patch completion based on the target pose human body mesh, and transfer the linear hybrid skin weights to the target pose human body mesh to obtain a driveable human body mesh. The texture optimization module is used to backproject the colors of the multi-view human body image onto the texture space of the driveable human body mesh to obtain a texture mesh.

10. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the human body mesh reconstruction method as described in any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, Used to store computer programs; wherein the computer programs, when executed by a processor, implement the human body mesh reconstruction method as described in any one of claims 1 to 8.

Images