Method and device for constructing controllable three-dimensional digital human generation model

Abstract

The invention provides a method and a device for constructing a controllable three-dimensional digital human generative model. The method comprises the following steps: acquiring human body point clouds under different postures; training a deformation model and an occupancy field network in a controllable digital human generation model by using the human body point cloud, and obtaining a second human body grid based on a standard space by using the trained deformation model and occupancy field network; and training a color prediction network in the controllable digital human generation model by using the second human body grid, and finally obtaining the trained controllable digital human generation model by using joint constraint. By training a small number of scanning human body grids with different postures, a high-quality virtual digital human model with textures can be generated, and the driven digital human model contains details such as non-rigid deformation and better conforms to a real physical deformation effect. And the driving speed can be generated in real time, so that the method has a wide application prospect in three-dimensional human body reconstruction.

Description

Technical Field

[0001] One or more embodiments of the present specification relate to the field of image processing technology, and in particular, to a method and device for constructing a controllable three-dimensional digital human generation model. Background Art

[0002] With the widespread application of virtual experience in games, social media, virtual reality (VR) and augmented reality (AR), there is a growing demand for realistic, personalized and animated virtual characters. As the digital avatars of users in virtual environments, these virtual characters play an important role in enhancing users' sense of participation and immersion. However, existing virtual character generation technologies mostly rely on fixed templates and limited control mechanisms, which are difficult to capture the subtle differences in human behavior and appearance, resulting in the generated virtual characters appearing rigid, single and lacking in realism, affecting the user experience.

[0003] Application Contents

[0004] This application describes a method and device for constructing a controllable three-dimensional digital human generation model, which can solve the above technical problems.

[0005] According to a first aspect, a method for constructing a controllable three-dimensional digital human generation model is provided, comprising:

[0006] Get human body point clouds in different postures;

[0007] After training the deformation model and the occupancy field network in the controllable digital human generation model using the human body point cloud, a second human body mesh based on the canonical space is obtained using the trained deformation model and the occupancy field network, wherein the second human body mesh includes vertices and faces;

[0008] The color prediction network in the controllable digital human generation model is trained using the second human body mesh, and the constraints of the color prediction network are generated using the deformation model, so as to finally obtain the trained controllable digital human generation model, which is used to output a three-dimensional digital human in an action space that meets the target action, target expression and target texture.

[0009] Based on the above embodiment, further, after using the human body point cloud to train the deformation model and the occupancy field network in the controllable digital human generation model, using the trained deformation model and the occupancy field network to obtain a second human body mesh based on the standard space specifically includes:

[0010] Obtaining human body point clouds and multi-view images in different postures, using the vertices extracted from the human body point clouds to train the forward process of the deformation model in the controllable digital human generation model, and using the trained forward process of the deformation model to obtain the vertices of the first human body mesh in the standard space, the multi-view image is based on the human body point cloud in one of the postures with the face facing forward in the standard space, and orthogonally projecting the scanned human body mesh from four angles of forward, reverse, left and right, respectively, to render the multi-view image;

[0011] Based on the vertices of the first human body mesh, an occupancy field network in the controllable digital human generation model is trained, and the trained occupancy field network is used to obtain a human body implicit geometric model, and a second human body mesh based on a standard space is extracted from the human body implicit geometric model by an isosurface value extraction method.

[0012] Based on the above embodiment, further, the method of using the second human body mesh to train the color prediction network in the controllable digital human generation model, and using the multi-view image and the deformation model to generate constraint conditions, and finally obtaining the trained controllable digital human generation model, specifically includes:

[0013] Using the second human body mesh to train the color prediction network in the controllable digital human generation model to obtain a third human body mesh, wherein the third human body mesh includes vertex colors and face colors;

[0014] The third human body mesh is input into the reverse process of the deformation model to obtain a fourth human body mesh based on the deformation space, wherein the fourth human body mesh conforms to the target action and the target expression, and based on the multi-view image and the fourth human body mesh, a joint constraint is determined so that the color prediction network generates the fourth human body mesh conforming to the target texture, and finally the trained controllable digital human generation model is obtained, and the controllable digital human generation model is used to output a three-dimensional digital human in an action space conforming to the target action, the target expression and the target texture.

[0015] Based on the above embodiment, further, the method of using the vertices extracted from the human body point cloud to train the forward process of the deformation model in the controllable digital human generation model, and using the trained forward process of the deformation model to obtain the vertices of the first human body mesh in the standard space, specifically includes:

[0016]

[0017] Iteratively solving the human body point cloud to obtain vertices for training, inputting the vertices for training into the reversible difference network in the deformation model, training the forward process of the reversible difference network, and using the occupancy field to query the results of the forward process output of the reversible difference network to obtain the vertices of the first human body mesh in the standard space;

[0018] where x d represents the vertex of the deformation space, that is, the vertex of the human body point cloud, x o represents the canonical space vertex that is independent of posture, Broy(., B) represents the iterative solution for obtaining the canonical space vertex, IDN represents the forward process of the reversible difference network, θ, ψ represent the posture parameter and expression parameter respectively, which are used to learn non-rigid deformation, Query(f occ (.)) means querying the vertex in the occupancy field.

[0019] Based on the above embodiment, further, the training of the occupancy field network in the controllable digital human generation model based on the vertices of the first human body mesh specifically includes:

[0020] The vertices of the first human body mesh are input into the occupancy field for training, and the predicted occupancy is output, and the mesh is extracted using the isosurface extraction method:

[0021]

[0022] f occ Represented as an occupancy field, implemented using a multilayer perceptron, with 0.5 being the grid surface area.

[0023] Based on the above embodiment, further, the method of using the second human body mesh to train the color prediction network in the controllable digital human generation model to obtain a third human body mesh specifically includes:

[0024] The second human body mesh is input into the color prediction network for training, and the color of the vertex is calculated by the input vertex position and normal. The color prediction is implemented as follows:

[0025] C = f color (Φ emb (V o , N o )),

[0026] Where C represents the vertex color, Φ emb (V o , N o ) means encoding vertices and normals, f color This is implemented using a multi-layer perceptron.

[0027] Based on the above embodiment, further, the reverse process of inputting the third human body mesh into the deformation model to obtain a fourth human body mesh based on the deformation space, the fourth human body mesh conforms to the target action and the target expression, and using the fourth human body mesh to generate constraint conditions so that the color prediction network generates the fourth human body mesh conforming to the target texture, specifically includes:

[0028]

[0029] Inputting the third human body mesh into the reversible difference network, executing the reverse process of the reversible difference network, and processing the output result by a linear skinning algorithm to obtain a fourth human body mesh in the deformation space;

[0030] Rendering the fourth human body mesh into a four-view image using a differentiable direction, and determining the joint constraint using the four-view image and the multi-view image so that the rendering texture of the four-view image is consistent with the target texture, wherein the joint constraint includes an absolute error constraint, a structural constraint, and a perceptual constraint;

[0031] IDN(- 1 )(., θ, ψ) represents the reverse process of the reversible differential network, LBS(., B) represents linear skinning, V d Represents a deformation space vertex.

[0032] According to a second aspect, there is provided a device for constructing a controllable three-dimensional digital human generation model, comprising:

[0033] The acquisition module is used to obtain the point cloud of the human body in different postures;

[0034] A first processing module is used to train a deformation model and an occupancy field network in a controllable digital human generation model using the human body point cloud, and then obtain a second human body mesh based on a standard space using the trained deformation model and the occupancy field network, wherein the second human body mesh includes vertices and faces;

[0035] The second processing module is used to train the color prediction network in the controllable digital human generation model using the second human body mesh, and finally obtain the trained controllable digital human generation model using joint constraints. The controllable digital human generation model is used to output a three-dimensional digital human in an action space that conforms to target actions, target expressions and target textures.

[0036] Based on the above embodiment, further, the first processing module is specifically used to obtain human body point clouds and multi-view images in different postures, use the vertices extracted from the human body point clouds to train the forward process of the deformation model in the controllable digital human generation model, and use the trained forward process of the deformation model to obtain the vertices of the first human body mesh in the standard space, and the multi-view image is based on the human body point cloud in one of the postures with the face facing forward in the standard space, and orthogonally projecting the scanned human body mesh from four angles of forward, reverse, left and right, respectively, to render the multi-view image;

[0037] Based on the vertices of the first human body mesh, an occupancy field network in the controllable digital human generation model is trained, and the trained occupancy field network is used to obtain a human body implicit geometric model, and a second human body mesh based on a standard space is extracted from the human body implicit geometric model by an isosurface value extraction method.

[0038] Based on the above embodiment, further, the second processing module is specifically used for:

[0039] Using the second human body mesh to train the color prediction network in the controllable digital human generation model to obtain a third human body mesh, wherein the third human body mesh includes vertex colors and face colors;

[0040] The third human body mesh is input into the reverse process of the deformation model to obtain a fourth human body mesh based on the deformation space, wherein the fourth human body mesh conforms to the target action and the target expression, and based on the multi-view image and the fourth human body mesh, a joint constraint is determined so that the color prediction network generates the fourth human body mesh conforming to the target texture, and finally the trained controllable digital human generation model is obtained, and the controllable digital human generation model is used to output a three-dimensional digital human in an action space conforming to the target action, the target expression and the target texture, and the joint constraint includes an absolute error constraint, a structural constraint and a perceptual constraint.

[0041] Based on the above embodiment, further, the first processing module is specifically used for:

[0042]

[0043] Iteratively solving the human body point cloud to obtain vertices for training, inputting the vertices for training into the reversible difference network in the deformation model, training the forward process of the reversible difference network, and using the occupancy field to query the results of the forward process output of the reversible difference network to obtain the vertices of the first human body mesh in the standard space;

[0044] where x d represents the vertex of the deformation space, that is, the vertex of the human body point cloud, x orepresents the canonical space vertex that is independent of posture, Broy(., B) represents the iterative solution for obtaining the canonical space vertex, IDN represents the forward process of the reversible difference network, θ, ψ represent the posture parameter and expression parameter respectively, which are used to learn non-rigid deformation, Query(f occ (.)) means querying the vertex in the occupancy field.

[0045] Based on the above embodiment, further, the first processing module is specifically used for:

[0046] The vertices of the first human body mesh are input into the occupancy field for training, and the predicted occupancy is output, and the mesh is extracted using the isosurface extraction method:

[0047]

[0048] f occ Represented as an occupancy field, implemented using a multilayer perceptron, with 0.5 being the grid surface area.

[0049] Based on the above embodiment, further, the second processing module is specifically used for:

[0050] The second human body mesh is input into the color prediction network for training, and the color of the vertex is calculated by the input vertex position and normal. The color prediction is implemented as follows:

[0051] C = f color (Φ emb (V o , N o )),

[0052] Where C represents the vertex color, Φ emb (V o , N o ) means encoding vertices and normals, f color This is implemented using a multi-layer perceptron.

[0053] Based on the above embodiment, further, the second processing module is specifically used for:

[0054]

[0055] Inputting the third human body mesh into the reversible difference network, executing the reverse process of the reversible difference network, and processing the output result by a linear skinning algorithm to obtain a fourth human body mesh in the deformation space;

[0056] Rendering the fourth human body mesh into a four-view image using a differentiable direction, and determining the joint constraint using the four-view image and the multi-view image so that the rendering texture of the four-view image is consistent with the target texture, wherein the joint constraint includes an absolute error constraint, a structural constraint, and a perceptual constraint;

[0057] IDN (-1) (., θ, ψ) represents the reverse process of the reversible differential network, LBS (., B) represents linear skinning, V d Represents a deformation space vertex.

[0058] According to a third aspect, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method for constructing a controllable three-dimensional digital human generation model as described in any one of the above technical solutions is implemented.

[0059] According to a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the method for constructing a controllable three-dimensional digital human generation model as described in any one of the above technical solutions is implemented.

[0060] In the above system and method provided in the embodiments of this specification, by training a small number of scanned human meshes in different postures, a high-quality, textured virtual digital human model can be generated, and the driven digital human model includes details such as non-rigid deformation, which is more in line with the real physical deformation effect. The driving speed can reach real-time generation, which makes the present invention have a wide range of application prospects in three-dimensional human reconstruction. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0062] Figure 1 A schematic diagram showing a flow chart of a method for constructing a controllable three-dimensional digital human generation model provided in an embodiment of this specification;

[0063] Figure 2 A schematic diagram showing a controllable three-dimensional digital human generation model provided in an embodiment of this specification;

[0064] Figure 3 A schematic diagram showing a flow chart of a method for constructing a controllable three-dimensional digital human generation model provided in an embodiment of this specification;

[0065] Figure 4A schematic diagram of a module of a device for constructing a controllable three-dimensional digital human generation model provided in an embodiment of this specification is shown. DETAILED DESCRIPTION

[0066] The solution provided in this specification is described below in conjunction with the accompanying drawings.

[0067] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0068] In the description of the embodiments of the present application, words such as "exemplary", "for example" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary", "for example" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary", "for example" or "for example" is intended to present related concepts in a concrete way.

[0069] In the description of the embodiments of the present application, the term "and / or" is merely a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B may represent: A exists alone, B exists alone, and A and B exist at the same time. In addition, unless otherwise specified, the term "plurality" means two or more.

[0070] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. The terms "include", "comprises", "has" and their variations all mean "including but not limited to", unless otherwise specifically emphasized.

[0071] In the field of computer graphics, the creation of digital humans usually involves two key steps: reconstruction and texture mapping. Early reconstruction work relied on displayed mesh models, which are suitable for existing graphics processing pipelines due to their topological consistency and precise geometric expression. However, these models have limitations in expressing human clothing and facial details. Subsequently, implicit models such as signed distance field (SDF) and occupancy field (OCC) were proposed to provide solutions for the expression of high-resolution details, but the problem of inconsistent mesh topology limits their application in other graphics pipelines. In terms of driving, human avatar deformation based on linear blend skinning (LBS) is the main way to drive the reconstructed avatar. In order to achieve non-rigid changes such as clothing wrinkles that are difficult to achieve with the LBS method, some methods add conditional parameters when calculating the implicit field to generate non-rigid deformations of different poses. However, this method requires the human mesh to be reconstructed separately for each pose, which not only leads to topological inconsistency, but also makes the mesh extraction process time-consuming, affecting the efficiency of digital human applications.

[0072] In terms of texture mapping, traditional methods achieve this by aligning color fields with 3D models, but this method requires a lot of data support to obtain high-precision human textures, and is difficult to achieve when the amount of data is limited. In recent years, techniques based on neural radiance fields (NeRF) and Gaussian splash (3DGS) have achieved remarkable results in rendering, and can reconstruct high-resolution human avatars that can be viewed from any perspective through multi-view videos. However, these rendering-based methods are usually unable to generate complete high-precision 3D human meshes, limiting their practicality in digital human-driven applications.

[0073] Therefore, how to provide a controllable human body generation method, system and storable medium that can generate expressive digital humans that can be efficiently driven by parameters is an urgent problem that needs to be solved by technicians in this field.

[0074] In view of the limitations of existing virtual character generation technology, this paper proposes an innovative framework to build expressive and highly customizable human virtual characters. It combines implicit geometric representation with explicit texture representation, effectively overcoming the shortcomings of traditional methods and achieving anatomical accuracy, detailed geometry, and fine control of clothing and surface details. Implicit geometric representation ensures that the virtual character maintains a natural body structure in real movement, and maintains anatomical accuracy and detailed geometry even in complex postures. Explicit texture representation provides users with advanced customization capabilities for clothing and surface details, making it easy to personalize style, color, and pattern, thereby creating unique and diverse virtual characters.

[0075] This paper introduces innovative reversible neural network technology to enhance the action expressiveness of virtual characters, solve the problem of stiff action, and ensure smooth and natural action. Based on the linear blend skinning (LBS) deformation, a reversible difference network (IDN) is integrated to learn the conditional dependent offset, and the pose-independent standard space grid is obtained by calculating the occupancy value of the offset standard space point cloud.

[0076] In the face of the challenges of detail capture and accurate texture mapping in 3D virtual character reconstruction, the present invention combines the advantages of implicit geometric structure with explicit texture representation. The canonical space human body mesh is extracted from the trained occupancy field through isosurface extraction technology, and then the color of the corresponding canonical space point cloud is predicted, and the point cloud is deformed to obtain the deformed space point cloud, thereby obtaining a posture-independent texture. Finally, in order to achieve high-precision texture, the present invention adopts multiple constraints to improve texture quality.

[0077] In summary, the present invention can quickly generate a 3D human model with high-precision textures and is easy to drive, providing users with a new virtual character generation solution. This solution not only improves generation efficiency, but also enhances the realism and personalized customization capabilities of virtual characters. Extensive experimental results and comprehensive evaluations further confirm the excellent performance of the present invention in generating digital human realism and driving efficiency.

[0078] like Figure 1 and Figure 2 As shown, in order to achieve the above purpose, the present invention adopts the following technical solutions:

[0079] Step (1): obtain the scanned human body meshes in different postures, process them into vertices in the deformation space, and render them into multi-view images;

[0080] Step (2): inputting the human body vertices in the deformation space into the deformation module, and after processing, obtaining the human body vertices in the standard space that are independent of the posture;

[0081] Step (3): input the vertices into the occupancy field network for training until the network converges, and then use the isosurface extraction algorithm to extract the human body mesh in the canonical space;

[0082] Step (4): input the human body mesh in the standard space into the color prediction network to calculate the color of each vertex, thereby obtaining a complete human body mesh with texture;

[0083] Step (5): input the human body mesh in the standard space into the deformation module, obtain the human body mesh in the deformation space, and render it into a multi-view image, applying multiple constraints to ensure that the texture of the rendering result is consistent with the target texture;

[0084] Step (6): Obtain the human body mesh in the canonical space and render it into multi-view images, edit these images, and use the edited images as target images and retrain until the network converges, thereby generating the edited texture.

[0085] Optionally, in step (1), the human face orientation in the standard space is taken as the positive orientation, and the scanned human body mesh is orthogonally projected from four angles, namely, the positive orientation, the negative orientation, the left orientation and the right orientation, to render a multi-view image.

[0086] Optionally, in step (1), vertex sampling is performed on the neighborhood area of ​​the scanned human body mesh, and the occupancy rate of each vertex is calculated, wherein the vertex occupancy rate inside the human body is 0, and the vertex occupancy rate outside the human body is 1.

[0087] Optionally, in step (2), during the training process, the deformation space vertices are converted into standard space vertices by an iterative root-finding method, and the standard space vertices are input into the forward process of the reversible difference network to obtain the standard space vertices that are independent of the posture, wherein the forward process of the reversible difference network is shown in the following formula:

[0088] x o =IDN(x c ,θ,ψ)=x c +Δx c ,

[0089] where x o represents the canonical space vertex independent of posture, IDN represents the forward process of the reversible difference network, θ, ψ represent the posture parameter and expression parameter respectively.

[0090] Optionally, the process of the deformation module in step (2) is as follows:

[0091]

[0092] where x d represents the vertex in the deformation space, x o represents the canonical space vertex that is independent of posture, Broy(., B) represents iterative solution, Query(f occ (.)) means querying the vertex in the occupancy field.

[0093] Optionally, in step (3), the vertices are input into the occupancy field for training, and the predicted occupancy is output, and then the mesh is extracted using an isosurface extraction method, wherein the occupancy field and isosurface are as follows:

[0094]

[0095] where 0.5 is defined as the mesh surface area, f occRepresented as an occupancy field, implemented using a multi-layer perceptron.

[0096] Optionally, in step (4), the color prediction network is expressed as follows:

[0097] C = f color (Φ emb (V o , N o )),

[0098] Where C represents the vertex color, Φ emb (V o , N o ) means encoding vertices and normals, f color This is implemented using a multi-layer perceptron.

[0099] Optionally, in step (5), the forward process of the deformation module is used to transform the posture-independent standard space vertices into deformation space vertices, and the specific process is as follows:

[0100]

[0101] IDN (-1) (., θ, ψ) represents the reverse process of the reversible differential network, LBS (., B) represents linear skinning, V d Represents a deformation space vertex.

[0102] Optionally, in step (5), a deformable spatial grid is rendered in a differentiable rendering manner to obtain the multi-view image.

[0103] Optionally, in step (5), the multiple constraints are expressed as follows:

[0104]

[0105] in represents the absolute error constraint, represents a structural constraint, Represents a perceived constraint.

[0106] Optionally, in step (6), any image editing tool is used to edit the human clothing in the rendered image, and then the color prediction network is retrained to obtain the edited texture.

[0107] In order to compare the method used in this paper with other high-performance human reconstruction driven algorithms in a quantitative manner, the performance of the state-of-the-art methods is shown: HAVE-FUN, Fast-SNARF, XAvatar, and the data and results are shown in Table 1.

[0108]

[0109] Table 1 Comparison of error and time spent on facial reconstruction

[0110] It can be seen from the above technical solutions that compared with the prior art, the present invention realizes efficient and high-quality digital human model generation through a series of innovative steps. First, the processed scanned human body mesh data is input into the inverse process of linear skinning to obtain the canonical space vertices, which enables the digital human model to be driven by any SMPLX parameters to achieve accurate simulation of any posture. Further, the canonical space vertices are input into the inverse process of the reversible difference network to obtain the canonical space vertices that are independent of posture. By learning non-rigid deformations, such as wrinkles and facial expressions, the expressive ability of the digital human is enhanced, and high-quality results can be obtained even under the training conditions of a small number of data sets, which significantly reduces the amount of data required for training. In addition, the canonical space vertices that are independent of posture are input into the occupancy field extraction grid, which effectively separates the deformation and human body reconstruction process, greatly improving the speed of digital human reconstruction. By inputting the extracted canonical space grid into the color field prediction vertex texture, the decoupling of texture and reconstruction is achieved, and high-quality texture is obtained while ensuring geometric quality. Finally, the human body mesh is rendered into a multi-view image using differentiable rendering technology, and combined with a variety of constraint methods based on two-dimensional images, the advantages of high-quality constraints and three-dimensional multi-view in two-dimensional images are fully utilized to generate realistic digital human textures. By training a small number of scanned human body meshes in different postures, a high-quality, textured virtual digital human model can be generated, and the driven digital human model contains details such as non-rigid deformation, which is more in line with the real physical deformation effect. The driving speed can reach real-time generation, which makes the present invention have broad application prospects in three-dimensional human body reconstruction. Extensive experimental results and comprehensive evaluations have confirmed the excellent performance of the present invention in texture quality and high efficiency of driving the human body in reconstructing three-dimensional human models.

[0111] Corresponding to the above method provided by the present invention, Figure 3 A flow chart showing a method for constructing a controllable three-dimensional digital human generation model provided in an embodiment of this specification includes:

[0112] 110. Obtain human body point clouds in different postures.

[0113] After using the human body point cloud to train the deformation model and the occupancy field network in the controllable digital human generation model, the trained deformation model and occupancy field network are used to obtain a second human body mesh based on the standard space, wherein the second human body mesh includes vertices and faces.

[0114] 130. The color prediction network in the controllable digital human generation model is trained by using the second human body mesh, and the trained controllable digital human generation model is finally obtained by using the joint constraint. The controllable digital human generation model is used to output a three-dimensional digital human in an action space that meets the target action, target expression and target texture.

[0115] Based on the above embodiment, further, step 120 specifically includes:

[0116] 121. Obtain human body point clouds and multi-view images in different postures, use the vertices extracted from the human body point clouds to train the forward process of the deformation model in the controllable digital human generation model, and use the forward process of the trained deformation model to obtain the vertices of the first human body mesh in the standard space. The multi-view image is based on the human body point cloud in one of the postures with the face facing forward in the standard space, and orthogonally projecting the scanned human body mesh from four angles of forward, reverse, left and right, respectively, to render the multi-view image;

[0117] 122 Based on the vertices of the first human body mesh, an occupancy field network in the controllable digital human generation model is trained, and the trained occupancy field network is used to obtain a human body implicit geometric model, and a second human body mesh based on the standard space is extracted from the human body implicit geometric model by an isosurface value extraction method.

[0118] Based on the above embodiment, further, step 130 specifically includes:

[0119] 131. Using the second human body mesh to train a color prediction network in the controllable digital human generation model to obtain a third human body mesh, wherein the third human body mesh includes vertex colors and face colors;

[0120] 132. Input the third human body mesh into the reverse process of the deformation model to obtain a fourth human body mesh based on the deformation space, the fourth human body mesh conforms to the target action and the target expression, and determine the joint constraints based on the multi-view image and the fourth human body mesh so that the color prediction network generates a fourth human body mesh that conforms to the target texture, and finally obtain the trained controllable digital human generation model, which is used to output a three-dimensional digital human in the action space that conforms to the target action, the target expression and the target texture.

[0121] Based on the above embodiment, further, step 121 specifically includes:

[0122]

[0123] Iteratively solving the human body point cloud to obtain vertices for training, inputting the vertices for training into the reversible difference network in the deformation model, training the forward process of the reversible difference network, and using the occupancy field to query the results of the forward process output of the reversible difference network to obtain the vertices of the first human body mesh in the standard space;

[0124] where x d represents the vertex of the deformation space, that is, the vertex of the human body point cloud, x o represents the canonical space vertex that is independent of posture, Broy(., B) represents the iterative solution for obtaining the canonical space vertex, IDN represents the forward process of the reversible difference network, θ, ψ represent the posture parameter and expression parameter respectively, which are used to learn non-rigid deformation, Query(f occ (.)) means querying the vertex in the occupancy field.

[0125] Based on the above embodiment, further, step 122 specifically includes:

[0126] The vertices of the first human mesh are fed into the occupancy field for training and the predicted occupancy is output, and the mesh is extracted using the isosurface extraction method:

[0127]

[0128] f occ Represented as an occupancy field, implemented using a multilayer perceptron, with 0.5 being the grid surface area.

[0129] Based on the above embodiment, further, step 131 specifically includes:

[0130] The second human body mesh is input into the color prediction network for training. The vertex color is calculated by inputting the vertex position and normal. The color prediction is implemented as follows:

[0131] C = f color (Φ emb (V o , N o )),

[0132] Where C represents the vertex color, Φ emb (V o , N o ) means encoding vertices and normals, f color This is implemented using a multi-layer perceptron.

[0133] Based on the above embodiment, further, step 132 specifically includes:

[0134]

[0135] The third human body mesh is input into the reversible difference network, the reverse process of the reversible difference network is executed, and the output result is processed by the linear skinning algorithm to obtain the fourth human body mesh in the deformation space;

[0136] The fourth human body mesh is rendered into a four-view image using a differentiable direction. The four-view image and the multi-view image are used to determine joint constraints so that the four-view image rendering texture is consistent with the target texture. The joint constraints include absolute error constraints, structural constraints and perceptual constraints.

[0137] IDN (-1) (., θ, ψ) represents the reverse process of the reversible difference network, LBS (., B) represents linear skinning, V d Represents a deformation space vertex.

[0138] It should be understood that the various constraints are expressed as follows:

[0139]

[0140] in represents the absolute error constraint, represents a structural constraint, Represents a perceived constraint.

[0141] Loss Function It is composed of the weighted sum of three different types of loss terms, which are the absolute error constraint Structural constraints and perceived constraints Each loss term is multiplied by a weight parameter, λ L1 ,λ ssim , and λ per These weight parameters are used to adjust the contribution of each loss term to the total loss. The entire summation expression is multiplied by It is used to adjust the overall size of the loss function which considers absolute error, structural similarity and perceptual quality during optimization.

[0142] The present invention combines implicit geometric representation with explicit texture representation, effectively overcomes the defects of traditional methods, and achieves anatomical accuracy, meticulous geometric structure, and fine control of clothing and surface details. Implicit geometric representation ensures that the virtual character maintains a natural body structure in real movement, and maintains anatomical accuracy and detailed geometric structure even in complex postures. Explicit texture representation provides users with advanced customization capabilities for clothing and surface details, easily realizing personalized selection of style, color and pattern, thereby creating unique and diverse virtual characters. Furthermore, innovative reversible neural network technology is introduced to enhance the action expressiveness of virtual characters, solve the problem of stiff movements, and ensure smooth and natural movements. Based on the linear blend skinning (LBS) deformation, a reversible differential network (IDN) is integrated to learn conditionally dependent offsets, and the occupancy value of the offset canonical space point cloud is calculated to obtain a canonical space grid that is independent of posture. In the face of the challenges of detail capture and accurate texture mapping in the reconstruction of three-dimensional virtual characters, the advantages of implicit geometric structure are combined with explicit texture representation. The canonical space human body mesh is extracted from the trained occupancy field through isosurface extraction technology, and then the color of the corresponding canonical space point cloud is predicted, and the point cloud is deformed to obtain the deformed space point cloud, thereby obtaining a posture-independent texture. Finally, in order to achieve high-precision texture, multiple constraints are used to improve the texture quality. The present invention can quickly generate a human body model with high-precision texture and easy to drive, providing users with a new virtual character generation solution. This solution not only improves the generation efficiency, but also enhances the realism and personalized customization capabilities of the virtual character. Extensive experimental results and comprehensive evaluations further confirm the excellent performance of the present invention in generating the realism and driving efficiency of digital humans.

[0143] Figure 4 It is shown that a device for constructing a controllable three-dimensional digital human generation model comprises:

[0144] The acquisition module is used to obtain the point cloud of the human body in different postures;

[0145] A first processing module is used to train a deformation model and an occupancy field network in a controllable digital human generation model using the human body point cloud, and then obtain a second human body mesh based on a standard space using the trained deformation model and the occupancy field network, wherein the second human body mesh includes vertices and faces;

[0146] The second processing module is used to train the color prediction network in the controllable digital human generation model using the second human body mesh, and finally obtain the trained controllable digital human generation model using joint constraints. The controllable digital human generation model is used to output a three-dimensional digital human in an action space that conforms to target actions, target expressions and target textures.

[0147] Based on the above embodiment, further, the first processing module is specifically used to obtain human body point clouds and multi-view images in different postures, use the vertices extracted from the human body point clouds to train the forward process of the deformation model in the controllable digital human generation model, and use the trained forward process of the deformation model to obtain the vertices of the first human body mesh in the standard space, and the multi-view image is based on the human body point cloud in one of the postures with the face facing forward in the standard space, and orthogonally projecting the scanned human body mesh from four angles of forward, reverse, left and right, respectively, to render the multi-view image;

[0148] Based on the vertices of the first human body mesh, an occupancy field network in the controllable digital human generation model is trained, and the trained occupancy field network is used to obtain a human body implicit geometric model, and a second human body mesh based on a standard space is extracted from the human body implicit geometric model by an isosurface value extraction method.

[0149] Based on the above embodiment, further, the second processing module is specifically used for:

[0150] Using the second human body mesh to train the color prediction network in the controllable digital human generation model to obtain a third human body mesh, wherein the third human body mesh includes vertex colors and face colors;

[0151] The third human body mesh is input into the reverse process of the deformation model to obtain a fourth human body mesh based on the deformation space, wherein the fourth human body mesh conforms to the target action and the target expression, and based on the multi-view image and the fourth human body mesh, a joint constraint is determined so that the color prediction network generates the fourth human body mesh conforming to the target texture, and finally the trained controllable digital human generation model is obtained, and the controllable digital human generation model is used to output a three-dimensional digital human in an action space conforming to the target action, the target expression and the target texture, and the joint constraint includes an absolute error constraint, a structural constraint and a perceptual constraint.

[0152] Based on the above embodiment, further, the first processing module is specifically used for:

[0153]

[0154] Iteratively solving the human body point cloud to obtain vertices for training, inputting the vertices for training into the reversible difference network in the deformation model, training the forward process of the reversible difference network, and using the occupancy field to query the results of the forward process output of the reversible difference network to obtain the vertices of the first human body mesh in the standard space;

[0155] where x d represents the vertex of the deformation space, that is, the vertex of the human body point cloud, x orepresents the canonical space vertex that is independent of posture, Broy(., B) represents the iterative solution for obtaining the canonical space vertex, IDN represents the forward process of the reversible difference network, θ, ψ represent the posture parameter and expression parameter respectively, which are used to learn non-rigid deformation, Query(f occ (.)) means querying the vertex in the occupancy field.

[0156] Based on the above embodiment, further, the first processing module is specifically used for:

[0157] The vertices of the first human body mesh are input into the occupancy field for training, and the predicted occupancy is output, and the mesh is extracted using the isosurface extraction method:

[0158]

[0159] f occ Represented as an occupancy field, implemented using a multilayer perceptron, with 0.5 being the grid surface area.

[0160] Based on the above embodiment, further, the second processing module is specifically used for:

[0161] The second human body mesh is input into the color prediction network for training, and the color of the vertex is calculated by the input vertex position and normal. The color prediction is implemented as follows:

[0162] C = f color (Φ emb (V o , N o )),

[0163] Where C represents the vertex color, Φ emb (V o , N o ) means encoding vertices and normals, f color This is implemented using a multi-layer perceptron.

[0164] Based on the above embodiment, further, the second processing module is specifically used for:

[0165]

[0166] Inputting the third human body mesh into the reversible difference network, executing the reverse process of the reversible difference network, and processing the output result by a linear skinning algorithm to obtain a fourth human body mesh in the deformation space;

[0167] Rendering the fourth human body mesh into a four-view image using a differentiable direction, and determining the joint constraint using the four-view image and the multi-view image so that the rendering texture of the four-view image is consistent with the target texture, wherein the joint constraint includes an absolute error constraint, a structural constraint, and a perceptual constraint;

[0168] IDN (-1) (., θ, ψ) represents the reverse process of the reversible differential network, LBS (., B) represents linear skinning, V d Represents a deformation space vertex.

[0169] Represented as an occupancy field, implemented using a multilayer perceptron, according to a third aspect, an electronic device is provided, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein when the processor executes the program, a method for constructing a controllable three-dimensional digital human generation model as described in any one of the above technical solutions is implemented.

[0170] According to a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the method for constructing a controllable three-dimensional digital human generation model as described in any one of the above technical solutions is implemented.

[0171] Those skilled in the art should be aware that in one or more of the above examples, the functions described in this application can be implemented by hardware, software, firmware, or any combination thereof. When implemented by software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium.

[0172] The specific implementation methods described above further illustrate the purpose, technical solutions and beneficial effects of the present application in detail. It should be understood that the above description is only the specific implementation methods of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent substitutions, improvements, etc. made on the basis of the technical solutions of the present application should be included in the scope of protection of the present application.

Claims

1. A method for constructing a controllable three-dimensional digital human generation model, characterized in that: include; Get human body point clouds in different postures; After training the deformation model and the occupancy field network in the controllable digital human generation model using the human body point cloud, a second human body mesh based on the canonical space is obtained using the trained deformation model and the occupancy field network, wherein the second human body mesh includes vertices and faces; The color prediction network in the controllable digital human generation model is trained using the second human body mesh, and the trained controllable digital human generation model is finally obtained using joint constraints. The controllable digital human generation model is used to output a three-dimensional digital human in an action space that meets the target action, target expression and target texture.

2. The method according to claim 1, characterized in that After using the human body point cloud to train the deformation model and the occupancy field network in the controllable digital human generation model, using the trained deformation model and the occupancy field network to obtain a second human body mesh based on the standard space, specifically includes: Obtaining human body point clouds and multi-view images in different postures, using the vertices extracted from the human body point clouds to train the forward process of the deformation model in the controllable digital human generation model, and using the trained forward process of the deformation model to obtain the vertices of the first human body mesh in the standard space, the multi-view image is based on the human body point cloud in one of the postures with the face facing forward in the standard space, and orthogonally projecting the scanned human body mesh from four angles of forward, reverse, left and right, respectively, to render the multi-view image; Based on the vertices of the first human body mesh, an occupancy field network in the controllable digital human generation model is trained, and the trained occupancy field network is used to obtain a human body implicit geometric model, and a second human body mesh based on a standard space is extracted from the human body implicit geometric model by an isosurface value extraction method.

3. The method according to claim 2, characterized in that The method of using the second human body mesh to train the color prediction network in the controllable digital human generation model and using joint constraints to finally obtain the trained controllable digital human generation model specifically includes: Using the second human body mesh to train the color prediction network in the controllable digital human generation model to obtain a third human body mesh, wherein the third human body mesh includes vertex colors and face colors; The third human body mesh is input into the reverse process of the deformation model to obtain a fourth human body mesh based on the deformation space, wherein the fourth human body mesh conforms to the target action and the target expression, and based on the multi-view image and the fourth human body mesh, a joint constraint is determined so that the color prediction network generates the fourth human body mesh conforming to the target texture, and finally the trained controllable digital human generation model is obtained, and the controllable digital human generation model is used to output a three-dimensional digital human in an action space conforming to the target action, the target expression and the target texture.

4. The method according to claim 2, characterized in that: The method of using the vertices extracted from the human body point cloud to train the forward process of the deformation model in the controllable digital human generation model, and using the trained forward process of the deformation model to obtain the vertices of the first human body mesh in the standard space, specifically includes: Iteratively solving the human body point cloud to obtain vertices for training, inputting the vertices for training into the reversible difference network in the deformation model, training the forward process of the reversible difference network, and using the occupancy field to query the results of the forward process output of the reversible difference network to obtain the vertices of the first human body mesh in the standard space; where x d represents the vertex of the deformation space, that is, the vertex of the human body point cloud, x o represents the canonical space vertex that is independent of posture, Broy(.,B) represents the iterative solution for obtaining the canonical space vertex, IDN represents the forward process of the reversible difference network, θ, ψ represent the posture parameter and expression parameter respectively, which are used to learn non-rigid deformation, Query(f occ (.)) indicates querying a vertex in the occupancy field.

5. The method according to claim 4, characterized in that The training of the occupancy field network in the controllable digital human generation model based on the vertices of the first human body mesh specifically includes: The vertices of the first human body mesh are input into the occupancy field for training, and the predicted occupancy is output, and the mesh is extracted using the isosurface extraction method: f occ Represented as an occupancy field, implemented using a multilayer perceptron, with 0.5 being the grid surface area.

6. The method according to claim 3, characterized in that The method of using the second human body mesh to train the color prediction network in the controllable digital human generation model to obtain a third human body mesh specifically includes: The second human body mesh is input into the color prediction network for training, and the color of the vertex is calculated by the input vertex position and normal. The color prediction is implemented as follows: C=f color (Φ emb (V o ,N o )), Where C represents the vertex color, Φ emb (V o ,N p ) means encoding vertices and normals, f color This is implemented using a multi-layer perceptron.

7. The method according to claim 6, characterized in that The inverse process of inputting the third human body mesh into the deformation model, obtaining a fourth human body mesh based on the deformation space, the fourth human body mesh conforming to the target action and the target expression, and determining a joint constraint based on the multi-view image and the fourth human body mesh so that the color prediction network generates the fourth human body mesh conforming to the target texture, specifically includes: Inputting the third human body mesh into the reversible difference network, executing the reverse process of the reversible difference network, and processing the output result by a linear skinning algorithm to obtain a fourth human body mesh in the deformation space; Rendering the fourth human body mesh into a four-view image using a differentiable direction, and determining the joint constraint using the four-view image and the multi-view image so that the rendering texture of the four-view image is consistent with the target texture, wherein the joint constraint includes an absolute error constraint, a structural constraint, and a perceptual constraint; IDN (-1) (.,θ,ψ) represents the reverse process of the reversible difference network, LBS (.,B) represents linear skinning, V d Represents a deformation space vertex.

8. A device for constructing a controllable three-dimensional digital human model, characterized in that: include; The acquisition module is used to obtain the point cloud of the human body in different postures; A first processing module is used to train a deformation model and an occupancy field network in a controllable digital human generation model using the human body point cloud, and then obtain a second human body mesh based on a standard space using the trained deformation model and the occupancy field network, wherein the second human body mesh includes vertices and faces; The second processing module is used to train the color prediction network in the controllable digital human generation model using the second human body mesh, and generate constraints of the color prediction network using the deformation model, and finally obtain the trained controllable digital human generation model, and the controllable digital human generation model is used to output a three-dimensional digital human in an action space that meets the target action, target expression and target texture.

9. An electronic device, characterized in that: The invention comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, a method for constructing a controllable three-dimensional digital human generation model as claimed in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor, the method for constructing a controllable three-dimensional digital human generation model as claimed in any one of claims 1 to 7 is implemented.

Images