Super-realistic digital human generation method, storage medium, product and equipment
By generating primitive face static mesh and texture maps, using standard mesh for topological reconstruction and bone binding, we automatically generate target face skeletal mesh, solving the problem of cumbersome production process of traditional digital humans and achieving efficient and controllable digital human generation.
Patent Information
- Application Number
- CN202510547006.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-04-28
- Publication Date
- 2025-08-05
AI Technical Summary
The traditional digital man production process is cumbersome, resulting in inefficient generation, consuming a lot of labor and time costs, and the model effect depends on the technical level of technicians, making it difficult to uniformly standardize.
The original face static mesh and texture map are generated based on the target face, topological reconstruction and bone binding are performed through preset standard mesh, and the target face skeleton mesh is automatically generated and rendered, simplifying the production process.
It improves the efficiency of digital life generation, reduces labor and time costs, ensures that the model effect is controllable, and does not rely on the technical level of technicians.
Smart Images
Figure CN120431235A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of three-dimensional models, and in particular to a method for generating a hyper-realistic digital human, a computer-readable storage medium, a computer program product, and a terminal device. Background Art
[0002] With the rapid development of network computing power and generative artificial intelligence AIGC (Artificial Intelligence Generated Content), and the acceleration of the digitalization process, there is a huge demand for digital people in the fields of games, film and television, live broadcasting, industry, advertising, etc.
[0003] The traditional digital human production process is as follows: (1) In 3D sculpting software (e.g., ZBrush, 3D coat, MudBox, etc.), sculpt the high-poly model of the digital human head and body required for the target face; (2) In 3D software (e.g., Maya, 3Dmax, Blender, etc.), retopologize the low-poly model and perform UV unfolding operations, then bake the required textures in baking software (e.g., Marmoset Toolbag, Substance Painter, etc.), and draw color maps and material maps in texture drawing software; (3) Perform bone binding and skinning in 3D software, create facial Blendshapes, and finally manually add the corresponding animation and import it into the engine for rendering.
[0004] However, the traditional digital human production process is cumbersome. Each digital human requires a lot of time to finely sculpt the facial structure, add facial details and textures from scratch, and re-bind the bones, create facial expressions and perform blendshapes. At the same time, the traditional digital human production process also requires a lot of manual intervention, which not only leads to low digital human generation efficiency, but also consumes a lot of labor and time costs. Furthermore, the effect of the generated digital human model depends on the technical level of the technicians, making it difficult to unify production standards, and also makes the digital human model effect uncontrollable. Summary of the Invention
[0005] The purpose of the embodiments of the present invention is to provide a method for generating a hyper-realistic digital human, a computer-readable storage medium, a computer program product, and a terminal device, which can improve the efficiency of digital human generation, significantly reduce the labor and time costs of generating digital humans, and ensure that the effects of the generated digital human models are controllable.
[0006] To achieve the above objectives, an embodiment of the present invention provides a method for generating a hyper-realistic digital human, comprising:
[0007] Generate an original face static mesh and original facial texture map based on the target face;
[0008] Retopologizing the original face static mesh according to a preset standard mesh to obtain a target face static mesh;
[0009] Generate various maps of the target face static mesh according to the original facial texture map and the target face static mesh;
[0010] Transferring the skeleton binding information of the standard mesh to the target face static mesh to generate a target face skeletal mesh;
[0011] The target human face skeleton mesh and the maps are rendered to obtain a target hyper-realistic digital human.
[0012] Furthermore, generating an original face static mesh and an original facial texture map based on the target face specifically includes:
[0013] When the target face does not participate in the light field scanning, collecting multiple photos corresponding to the target face at multiple angles;
[0014] The multiple photos are input into a preset 3D reconstruction software, and a static mesh preset in the 3D reconstruction software is used to perform multi-angle feature matching with the multiple photos to perform 3D face reconstruction, and texture coordinate mapping is performed to obtain the original face static mesh and the original facial texture map that matches the original face static mesh.
[0015] Furthermore, generating an original face static mesh and an original facial texture map based on the target face specifically includes:
[0016] When the target face participates in light field scanning, the target face is scanned at multiple angles and multiple positions by a light field scanning device to obtain light information and color information corresponding to the target face;
[0017] Reconstructing the light information to obtain three-dimensional coordinates of facial feature points of the target face;
[0018] A texture map is generated according to the color information, and the generated texture map is combined with the three-dimensional coordinates to obtain an original human face static mesh; wherein the original human face static mesh contains the original facial texture map.
[0019] Furthermore, retopologically re-forming the original face static mesh according to a preset standard mesh to obtain a target face static mesh specifically includes:
[0020] Inputting the preset standard mesh and the original face static mesh into the preset retopology software;
[0021] The retopology software is used to mark corresponding points on the facial feature points of the two meshes in sequence and then perform wrapping deformation so that the facial feature points of the standard mesh match the facial feature points of the original face static mesh, thereby obtaining the target face static mesh.
[0022] Furthermore, generating various maps of the target face static mesh based on the original facial texture map and the target face static mesh specifically includes:
[0023] Performing normal baking on the target face static mesh, and baking and transferring the original facial texture map to the target face static mesh, thereby obtaining a normal map and a color map;
[0024] The baked normal map and color map are repaired according to the normal map and color map of the standard mesh, and the repaired normal map and color map are used as the normal map and color map of the target face static mesh.
[0025] Furthermore, transferring the skeleton binding information of the standard mesh to the target face static mesh to generate the target face skeleton mesh specifically includes:
[0026] Input the standard mesh and the target face static mesh into DCC software;
[0027] Calculate the bone offset corresponding to each bone node in the target face static mesh according to the bone binding information and vertex coordinates of the standard mesh and the vertex coordinates of the target face static mesh;
[0028] Move the bone position of each bone node in the target face static mesh according to the bone offset corresponding to each bone node;
[0029] The target face static mesh is re-bound to the relocated bone nodes, and the bone weights of the standard mesh are transferred to the target face static mesh using the transfer function in the DCC software to generate a target face skeletal mesh.
[0030] Furthermore, the step of calculating the bone offset corresponding to each bone node in the target face static mesh according to the bone binding information and vertex coordinates of the standard mesh and the vertex coordinates of the target face static mesh specifically includes:
[0031] For each bone node in the target face static mesh, determine a target bone node corresponding to the bone node according to the bone binding information of the standard mesh;
[0032] Determine a first target vertex closest to the target bone node from all vertices of the standard mesh;
[0033] Determine, based on the first target vertex, a second target vertex corresponding to the first target vertex from all vertices of the target face static mesh;
[0034] The bone offset corresponding to the bone node is calculated based on the center coordinates of the target bone node, the center coordinates of the first target vertex, and the center coordinates of the second target vertex.
[0035] An embodiment of the present invention further provides a computer-readable storage medium, comprising a stored computer program, wherein when the computer program is run, the device where the computer-readable storage medium is located controls the device to execute any of the above-mentioned methods for generating a hyper-realistic digital human.
[0036] An embodiment of the present invention further provides a computer program product, including a computer program, which, when executed by a processor, implements any of the above-mentioned methods for generating a hyper-realistic digital human.
[0037] An embodiment of the present invention further provides a terminal device comprising a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor implements any of the above-described methods for generating a hyper-realistic digital human when executing the computer program.
[0038] Compared to the prior art, the embodiments of the present invention provide a method for generating a hyper-realistic digital human, a computer-readable storage medium, a computer program product, and a terminal device, including: generating an original face static mesh and an original facial texture map based on a target face; retopologically ... BRIEF DESCRIPTION OF THE DRAWINGS
[0039] Figure 1 This is a flow chart of a preferred embodiment of a method for generating a hyper-realistic digital human provided by the present invention;
[0040] Figure 2 This is an example diagram of generating an original human face static mesh and an original facial texture map provided by an embodiment of the present invention;
[0041] Figure 3 This is an example diagram of generating a target face static mesh according to an embodiment of the present invention;
[0042] Figure 4 This is an example diagram of generating a facial map of a target face provided by an embodiment of the present invention;
[0043] Figure 5 This is an example diagram of generating a target face skeletal mesh according to an embodiment of the present invention;
[0044] Figure 6 It is a structural block diagram of a preferred embodiment of a terminal device provided by the present invention. DETAILED DESCRIPTION
[0045] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this technical field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.
[0046] The embodiment of the present invention provides a method for generating a hyper-realistic digital human. Figure 1 FIG. 1 is a flow chart of a preferred embodiment of a method for generating a hyper-realistic digital human provided by the present invention, wherein the method comprises steps S11 to S15:
[0047] Step S11: Generate an original face static mesh and an original facial texture map based on the target face;
[0048] Step S12: retopologically re-topologizing the original face static mesh according to a preset standard mesh to obtain a target face static mesh;
[0049] Step S13: Generate various maps of the target face static mesh based on the original facial texture map and the target face static mesh;
[0050] Step S14: Transfer the skeleton binding information of the standard mesh to the target face static mesh to generate a target face skeleton mesh;
[0051] Step S15: Render the target face skeleton mesh and the various maps to obtain a target hyper-realistic digital human.
[0052] Specifically, first, an original face static mesh and an original facial texture map that matches the original face static mesh are generated based on the target face (referring to the real face of the target). Since the original face static mesh generated at this time does not contain facial binding or does not meet the facial binding specifications (for example, the mesh generated by light field scanning in the following embodiment is mostly triangular faces), it is necessary to retopologize the generated original face static mesh with facial binding. In an embodiment of the present invention, the generated original face static mesh can be retopologized according to a preset standard mesh (containing facial binding) to obtain a target face static mesh with facial binding. Then, baking mapping and repair processing are performed based on the obtained original facial texture map and the target face static mesh to generate various texture maps of the target face static mesh. At this time, the static mesh of the target face has been generated. Therefore, it is necessary to generate corresponding skeletal nodes. Since the pre-set standard mesh contains facial binding, and based on the above-mentioned topological reconstruction, the corresponding target face static mesh is consistent with the topological structure of the pre-set standard mesh. Therefore, in an embodiment of the present invention, the facial bone binding information of the pre-set standard mesh can be transferred to the target face static mesh through bone skinning for reuse, so that the bones of the pre-set standard mesh can successfully drive the target face static mesh, and a target face skeletal mesh with facial bone binding is obtained accordingly; finally, the texture maps of the obtained target face skeletal mesh and the target face static mesh are input into a pre-set three-dimensional rendering engine, the corresponding material balls and texture maps are set, and the three-dimensional rendering engine is used for rendering output to obtain the target hyper-realistic digital human corresponding to the target face.
[0053] It should be noted that the pre-set standard meshes can be flexibly selected by relevant technical personnel. For example, the MetaHuman basic mesh or the Character Creator basic mesh can be selected. These basic meshes have uniform wiring, can meet the standards of facial animation, and have preset a large number of Blendshapes or facial bone binding information for users to use.
[0054] It should be noted that a static mesh is a basic unit for creating geometric bodies in a virtual scene. It is a piece of geometry composed of a set of polygons. These polygons can be cached in the video memory and rendered using the graphics card. "Static" means that the type of the mesh itself is static and does not contain dynamic movement or changing animation information, but does not mean that the mesh cannot move. For example, a static mesh can be a three-dimensional model created in a third-party modeling software (such as the original human face static mesh in the embodiment of the present invention). Some maps used in the game also contain static meshes. In addition, static meshes can also include movable objects such as doors or elevators, as well as many visual elements such as buildings, trees and terrain decorations.
[0055] In the embodiment of the present invention, the "original face static mesh" is a static mesh corresponding to the target real face obtained by directly creating it in a third-party modeling software using the target's real face; the "standard mesh" is a basic mesh, which is the basic element used in the software to construct a complex model; the "target face static mesh" is another static mesh corresponding to the target real face obtained by retopologically ... The process of converting an original human face static mesh (such as the target human face static mesh in the embodiment of the present invention) into a smaller model that can be used for animation (such as the target human face static mesh in the embodiment of the present invention) is to overlay the low polygon mesh onto the high polygon mesh to obtain a polygon mesh that is easier to use for animation; facial binding refers to the binding of facial bone data, which is a key part of modeling work and an important process of animation production; therefore, the embodiment of the present invention uses a standard mesh to retopologize the original human face static mesh, so that the facial bone data can be more easily bound to the original human face static mesh, thereby obtaining a more effective target human face static mesh.
[0056] It should be noted that skeletons define the skeletal structure of a character or model, which includes joints (bones) and a hierarchy of bones connecting these joints. Each joint typically has its own displacement, rotation, and scale information relative to the parent joint. Skeletons are the foundation of model animation, determining the character's skeleton and the hierarchy of actions. Binding refers to binding a mesh to the skeleton hierarchy of bones. This allows the bones to affect the mesh's vertices, causing the mesh to deform when the bones move. Skeletal binding is used to drive the mesh through the movement of the bones.
[0057] It should be noted that a skeletal mesh is an appearance representation of a character or model. It is combined with bones to produce animation effects and can be used to simulate the movements and morphological changes of three-dimensional objects. It does this by dividing an object into multiple small mesh units and embedding a series of bones inside each unit. Mathematical operations are then used to calculate the relationship between these bones, thereby achieving various deformations of the object, such as bending, rotating, and stretching. At runtime, the animation system affects the appearance of the skeletal mesh by modifying the transformation of the bones, thereby presenting the movement of the model. Compared to static meshes, skeletal meshes are more powerful. They are composed of information such as vertices, normals, and texture coordinates. Compared to static meshes, they add three-dimensional information such as bones, skeletal animation, and deformation, and are most commonly used for information carrying in three-dimensional models. For example, in the fields of movies and games, by using skeletal mesh technology, filmmakers and game developers can achieve realistic object animation effects, allowing characters to show vivid skeletal movements during movement, and also achieve more natural object morphological changes.
[0058] In the embodiment of the present invention, the "target face skeletal mesh" is a skeletal mesh corresponding to the target real face obtained by driving the target face static mesh using a standard mesh. Compared with the "target face static mesh", the "target face skeletal mesh" contains facial bone binding information (i.e., binding information of facial bone data) used to drive the skeletal mesh.
[0059] A method for generating a hyper-realistic digital human provided by an embodiment of the present invention automatically achieves the correspondence between facial feature points of the standard mesh and the original static mesh by retopologically retopologically using the topological structure of a standard mesh. Furthermore, the skeletal binding information of the standard mesh is transferred to the static mesh of a target face using skeletal skinning, automatically reusing the skeletal binding information of the target static mesh and the standard mesh. This simplifies the digital human production process, eliminates the need for extensive manual intervention, and enables more efficient generation of digital humans in different scenarios. This effectively improves the efficiency of digital human generation and significantly reduces the labor and time costs of generating digital humans. Furthermore, the effect of the generated digital human model does not need to rely on the technical level of the technician, ensuring that the effect of the generated digital human model is controllable.
[0060] In one optional embodiment, generating an original face static mesh and an original facial texture map based on the target face specifically includes:
[0061] When the target face does not participate in the light field scanning, collecting multiple photos corresponding to the target face at multiple angles;
[0062] The multiple photos are input into a preset 3D reconstruction software, and a static mesh preset in the 3D reconstruction software is used to perform multi-angle feature matching with the multiple photos to perform 3D face reconstruction, and texture coordinate mapping is performed to obtain the original face static mesh and the original facial texture map that matches the original face static mesh.
[0063] Specifically, in combination with the above embodiments, when generating the original face static mesh and the original facial texture map based on the target face, it is possible to first determine whether the target face can participate in the light field scan. If it is determined that the target face cannot participate in the light field scan, the original face static mesh and the original facial texture map can be generated through the photo of the target face. The specific implementation is as follows: collect multiple photos corresponding to the target face at multiple different angles (only including the three angles of front, side, and 45° side), and input the collected multiple photos into a pre-set three-dimensional reconstruction software for three-dimensional face reconstruction, and obtain the original face static mesh of the target face. At the same time, the texture coordinates of the target face in the photo are mapped in the pre-set three-dimensional reconstruction software, and the original facial texture map matching the original face static mesh is obtained.
[0064] Furthermore, the step of inputting the plurality of photos into a preset 3D reconstruction software to perform 3D face reconstruction specifically includes:
[0065] The multiple photos are input into a preset three-dimensional reconstruction software, and a static mesh preset in the three-dimensional reconstruction software is used to perform multi-angle feature matching with the multiple photos to perform three-dimensional face reconstruction.
[0066] Specifically, in combination with the above embodiments, when actually performing three-dimensional face reconstruction, the multiple collected photos can be input into a pre-set three-dimensional reconstruction software, and the static mesh pre-set in the three-dimensional reconstruction software can be used to match multi-angle features (for example, eyebrows, eyes, nose, mouth, face shape, etc.) with the multiple photos before performing three-dimensional face reconstruction.
[0067] It should be noted that feature points that need to be matched, such as eyebrows, eyes, nose, mouth, and face shape, can be detected using computer vision algorithms. For example, feature point detection can be performed using Dlib, OpenCV, or a deep learning-based model.
[0068] It should be noted that texture mapping is a technology that maps image information onto the mesh of an object, thereby increasing the details of the object's surface and making the object more realistic; texture coordinates represent the image information that needs to be mapped. For example, the image information is the pixel information of the image, and the texture coordinates can be represented by the pixel matrix of the image. For example, the three-dimensional texture coordinates can be a three-dimensional pixel matrix in the form of (x, y, z). Correspondingly, three-dimensional texture mapping is to map this three-dimensional pixel array onto the mesh of the object.
[0069] In the embodiment of the present invention, the "original facial texture map" is the facial texture map corresponding to the target real face, that is, the facial texture coordinates of the target real face are mapped to the original face static mesh through texture coordinate mapping.
[0070] See also Figure 2 , which is an example diagram of generating an original human face static mesh and an original facial texture map according to an embodiment of the present invention. Figure 2 On the left are multiple photos of the target face taken at different angles. Figure 2 On the right is the original face static mesh and original facial texture map generated after 3D face reconstruction in 3D reconstruction software ( Figure 2 The pictures shown are grayscale, the actual pictures should be in color and Figure 2 The real face on the left is blurred).
[0071] In one optional embodiment, generating an original face static mesh and an original facial texture map based on the target face specifically includes:
[0072] When the target face participates in light field scanning, the target face is scanned at multiple angles and multiple positions by a light field scanning device to obtain light information and color information corresponding to the target face;
[0073] Reconstructing the light information to obtain three-dimensional coordinates of facial feature points of the target face;
[0074] A texture map is generated according to the color information, and the generated texture map is combined with the three-dimensional coordinates to obtain an original human face static mesh; wherein the original human face static mesh contains the original facial texture map.
[0075] Specifically, in combination with the above embodiments, when generating an original face static mesh and an original facial texture map based on a target face, it is possible to first determine whether the target face can participate in light field scanning. If it is determined that the target face can participate in light field scanning, the target face can be subjected to light field scanning to generate an original face static mesh and an original facial texture map. The specific implementation is as follows: the scanned target face is placed in an environment with appropriate lighting, and a light field scanning device (for example, a light field camera, etc.) is first used to scan the target face at multiple different angles and multiple different positions to cover the entire target face surface, and the light information and color information corresponding to the target face are obtained accordingly; then, the three-dimensional coordinates of each facial feature point of the target face are reconstructed according to the light information obtained by the scan, and a corresponding texture map is generated according to the color information obtained by the scan, and the generated texture map is combined with the reconstructed three-dimensional coordinates to generate an original face static mesh, and the generated original face static mesh contains the original facial texture map.
[0076] It should be noted that when using a light field scanning device to scan the target face, the light field scanning device can not only capture traditional color information and brightness information, but also capture light information incoming from different directions (for example, the direction and intensity of the incident light), which allows the light field scanning device to obtain depth and normal information about each facial feature point during the capture process. Once this light information is captured, a computer algorithm can be used to reconstruct the three-dimensional coordinates of each facial feature point. In addition, the light field scanning device also captures the vertex color of the target face surface, and maps the vertex color to UV coordinates to generate a texture map, which is then combined with the three-dimensional coordinates to generate the original face static mesh containing the texture map.
[0077] In one optional embodiment, retopologically ...
[0078] Inputting the preset standard mesh and the original face static mesh into the preset retopology software;
[0079] The retopology software is used to mark corresponding points on the facial feature points of the two meshes in sequence and then perform wrapping deformation so that the facial feature points of the standard mesh match the facial feature points of the original face static mesh, thereby obtaining the target face static mesh.
[0080] Specifically, in combination with the above embodiment, when the original face static mesh is retopologically modified according to a preset standard mesh, the preset standard mesh containing facial binding and the original face static mesh can be input together into a preset retopology software (for example, Wrap4D or Houdini, etc.). The retopology software is used to mark corresponding points on the facial feature points of the two meshes (i.e., the standard mesh and the original face static mesh) in sequence and then perform wrapping deformation, so that the facial feature points of the standard mesh match the facial feature points of the original face static mesh without changing the topological structure, and the target face static mesh is obtained accordingly.
[0081] See also Figure 3 , which is an example diagram of generating a target face static mesh according to an embodiment of the present invention. Figure 3 On the left is the standard mesh with facial binding + the original face static mesh. Figure 3 The right side is the target face static mesh generated after retopology ( Figure 3 The images shown are in grayscale; actual images should be in color.)
[0082] It should be noted that the embodiment of the present invention imports the standard mesh and the original face static mesh into the retopology software at the same time, marks the feature points corresponding to the facial structures of the two meshes, and then performs wrapping deformation. By achieving the correspondence between the facial feature points corresponding to the standard mesh and the original face static mesh, it is convenient to use the facial feature points to find the corresponding bone nodes in the subsequent process of bone position adjustment (see the following embodiment for details).
[0083] In one optional embodiment, generating various maps of the target face static mesh based on the original facial texture map and the target face static mesh specifically includes:
[0084] Performing normal baking on the target face static mesh, and baking and transferring the original facial texture map to the target face static mesh, thereby obtaining a normal map and a color map;
[0085] The baked normal map and color map are repaired according to the normal map and color map of the standard mesh, and the repaired normal map and color map are used as the normal map and color map of the target face static mesh.
[0086] Specifically, in combination with the above embodiment, when generating various maps of the target face static mesh based on the original facial texture map and the target face static mesh, the target face static mesh can be re-normal-baked (i.e., baking the map is actually converting the light and shadow relationship between the models into a picture, thus forming a map), and the original face static mesh is imported into the baking software as the baked object, and the original facial texture map corresponding to the original face static mesh is baked and transferred to the target face static mesh to obtain a facial color texture map generated by a photo and conforming to the UV coordinates of a preset standard mesh. The baked normal map and color texture map are then superimposed and repaired with the normal map and color map of the preset standard mesh to obtain a repaired normal map and a repaired color map, and the repaired normal map is used as the normal map of the target face static mesh, and the repaired color map is used as the color map of the target face static mesh. When repairing, the image processing tool can be Photoshop, Substance Painter, etc., and can focus on repairing the normals and texture details of the eyes, nose and mouth.
[0087] See also Figure 4 FIG. 1 is an example diagram of generating a facial map of a target face provided by an embodiment of the present invention. Figure 4 On the left is the target face static mesh (including the original facial texture map). Figure 4 The four small pictures on the right are the normal map, color map and other target face maps obtained after normal baking and repair ( Figure 4 The images shown are in grayscale; actual images should be in color.)
[0088] The method for generating a hyper-realistic digital human provided by an embodiment of the present invention obtains various maps of the static mesh of the target face by utilizing the texture map of the original face and the texture map information of the standard mesh. This method automates the map production and eliminates the need for manual map drawing. This method can also simplify the digital human production process, effectively improve the efficiency of digital human generation, significantly reduce the labor and time costs of digital human generation, and ensure that the generated digital human model has a controllable effect.
[0089] It should be noted that since the pre-set standard mesh itself is already very accurate, relevant technicians only need to repair and optimize the assets provided by the standard mesh, which greatly reduces the threshold for digital human production. In addition, the standard mesh also contains multiple levels of LOD (Level of Detail), which can be reasonably selected according to the performance requirements of different platforms and project requirements. This makes the embodiment of the present invention applicable to different development environments and realizes the efficient production of hyper-realistic digital humans in different scenarios.
[0090] In one optional embodiment, transferring the skeleton binding information of the standard mesh to the target face static mesh to generate the target face skeletal mesh specifically includes:
[0091] Input the standard mesh and the target face static mesh into DCC software;
[0092] Calculate the bone offset corresponding to each bone node in the target face static mesh according to the bone binding information and vertex coordinates of the standard mesh and the vertex coordinates of the target face static mesh;
[0093] Move the bone position of each bone node in the target face static mesh according to the bone offset corresponding to each bone node;
[0094] The target face static mesh is re-bound to the relocated bone nodes, and the bone weights of the standard mesh are transferred to the target face static mesh using the transfer function in the DCC software to generate a target face skeletal mesh.
[0095] Specifically, in combination with the above embodiment, when the facial bone binding information of the preset standard mesh is transferred to the target face static mesh, the preset standard mesh with facial bone binding information and the target face static mesh can be imported into DCC software (for example, Maya, 3Dmax, Blender, etc.) respectively, and the bone offset corresponding to each bone node in the target face static mesh is calculated based on the facial bone binding information of the standard mesh, all vertex coordinates of the standard mesh (that is, the position coordinates of each vertex in the polygon corresponding to the mesh) and all vertex coordinates of the target face static mesh, and the bone offset corresponding to each bone node obtained is calculated based on the bone offset corresponding to each bone node. The bone position of each bone node in the face static mesh, and the movement amount of each bone node are the bone offsets corresponding to each bone node; then, the target face static mesh is re-bound to all the bone nodes after the position is moved, and the transfer function in the DCC software is used to transfer the bone weights of the standard mesh (that is, the weight setting of the constraint bone layer, through which the bone weights can control how the corresponding parts of the mesh follow the movement of the bones) to the target face static mesh, and the target face skeletal mesh is generated accordingly, thus completing the bone transfer of the target face static mesh, so that the pre-set standard mesh bones can successfully drive the target face static mesh, and the target face skeletal mesh with bone binding is obtained accordingly.
[0096] See also Figure 5, which is an example diagram of generating a target face skeleton mesh according to an embodiment of the present invention. Figure 5 The left side is the target face static mesh + the standard mesh with bone binding. Figure 5 The target face skeletal mesh obtained after the skeleton binding information is passed to the target face static mesh ( Figure 5 Using Unreal Engine's MetaHuman system mesh).
[0097] It should be noted that in the process of transferring the facial bone binding information of the preset standard mesh to the target human face static mesh, if the difference between the preset standard mesh and the target human face static mesh is too large, it may cause the Blendshapes modification of the preset standard mesh to be deformed. At this time, relevant technical personnel are required to repair the Blendshapes, and after the facial bone binding repair is completed, the facial bone binding information transfer process can continue.
[0098] In one optional embodiment, calculating the bone offset corresponding to each bone node in the target face static mesh based on the bone binding information and vertex coordinates of the standard mesh and the vertex coordinates of the target face static mesh specifically includes:
[0099] For each bone node in the target face static mesh, determine a target bone node corresponding to the bone node according to the bone binding information of the standard mesh;
[0100] Determine a first target vertex closest to the target bone node from all vertices of the standard mesh;
[0101] Determine, based on the first target vertex, a second target vertex corresponding to the first target vertex from all vertices of the target face static mesh;
[0102] The bone offset corresponding to the bone node is calculated based on the center coordinates of the target bone node, the center coordinates of the first target vertex, and the center coordinates of the second target vertex.
[0103] Specifically, in combination with the above embodiment, when actually calculating the bone offset corresponding to the bone node, due to the large number of vertices in the mesh body, it is necessary to use a loop to find the bone offset of each bone node, and the calculation method of the bone offset of each bone node is the same. Here, the calculation of the bone offset corresponding to any bone node (called the bone node to be calculated) in the static mesh of the target face is used as an example for explanation. The specific implementation is as follows: First, according to the preset bone binding information of the standard mesh body, the target bone node corresponding to the bone node to be calculated is selected from all the bone nodes of the standard mesh body, and the target bone node is selected according to the center coordinates of all the vertices of the standard mesh body and the target bone node. The center coordinates of the bone node are used to calculate the distance between each vertex of the standard mesh and the target bone node, so as to determine the vertex closest to the target bone node from all the vertices of the standard mesh as the first target vertex; then, based on the relationship of the same topological structure between the standard mesh and the target face static mesh, a vertex corresponding to the first target vertex is further determined from all the vertices of the target face static mesh according to the determined first target vertex as the second target vertex; finally, based on the center coordinates of the target bone node, the center coordinates of the first target vertex and the center coordinates of the second target vertex, the bone offset corresponding to the bone node to be calculated is calculated.
[0104] Furthermore, the step of calculating the bone offset corresponding to the bone node according to the center coordinates of the target bone node, the center coordinates of the first target vertex, and the center coordinates of the second target vertex specifically includes:
[0105] According to the formula Calculate and obtain the bone offset corresponding to the bone node; wherein, (BX, BY, BZ) represents the center coordinates of the target bone node, (M1X, M1Y, M1Z) represents the center coordinates of the first target vertex, (M2X, M2Y, M2Z) represents the center coordinates of the second target vertex, and (offsetX, offsetY, offsetZ) represents the bone offset corresponding to the bone node.
[0106] Specifically, in combination with the above embodiment, when calculating the bone offset corresponding to the bone node according to the center coordinates of the target bone node, the center coordinates of the first target vertex and the center coordinates of the second target vertex, assuming that the center coordinates of the target bone node are expressed as (BX, BY, BZ), the center coordinates of the first target vertex are expressed as (M1X, M1Y, M1Z), and the center coordinates of the second target vertex are expressed as (M2X, M2Y, M2Z), then it can be directly calculated according to the formula Calculate the bone offset (offsetX, offsetY, offsetZ) corresponding to the bone node.
[0107] For example, since the number of vertices in the mesh is large, it is necessary to loop to find the bone offset corresponding to each bone node, so it can be executed with a script command, specifically as follows: set M1 to represent the preset standard mesh, M2 to represent the target face static mesh, and Bone to represent the bone node; select a bone node with its center coordinates (BX, BY, BZ), calculate the distance between the coordinate positions of all vertices of M1 and the bone node, find the nearest vertex, and the center coordinate position of the vertex is represented by (M1X, M1Y, M1Z); since the vertex numbers corresponding to meshes with the same topological structure are the same, the vertex corresponding to the same vertex number of M2 can be found based on (M1X, M1Y, M1Z), and the center coordinate position of the vertex is represented by (M2X, M2Y, M2Z); then, the bone offset (offsetX, offsetY, offsetZ) can be obtained by the formula Calculated; by looping the above bone offset calculation steps, the bone offsets of all bone nodes can be obtained.
[0108] Understandably, traditional digital human facial high-poly model production requires a significant amount of time and effort to gradually refine and carve from scratch. While saving time, the embodiments of the present invention can also ensure more accurate and standardized production of high-precision digital humans. Furthermore, when performing skeletal rigging, high-quality skeletal rigging can be generated simply by automatically executing scripts, significantly improving the work efficiency of animation riggers.
[0109] In other optional embodiments, the method further includes:
[0110] After obtaining the target human face static mesh, the target human face static mesh is optimized using preset three-dimensional carving software.
[0111] Specifically, in combination with the above embodiment, after obtaining the target face static mesh containing facial binding through retopology, the obtained target face static mesh can be imported into a pre-set 3D engraving software, and the target face static mesh can be optimized using the 3D engraving software to obtain an optimized target face static mesh accordingly; wherein, during the actual optimization, the 3D character modeler can perform manual optimization in the 3D engraving software to repair perspective problems caused by photos or scanned data, and perform some engraving of facial details, such as eyelids, nasolabial folds, etc., to improve the accuracy of facial details and make the optimized target face static mesh more consistent with the target face.
[0112] It can be understood that after obtaining the optimized target face static mesh, baking mapping and repair processing can be performed based on the obtained original facial texture map and the optimized target face static mesh, and the various texture maps of the optimized target face static mesh are generated accordingly; the facial bone binding information of the pre-set standard mesh is passed to the optimized target face static mesh through bone skinning for reuse, and a target face skeletal mesh with bone binding is obtained accordingly; the obtained target face skeletal mesh and the various texture maps of the optimized target face static mesh are input into a pre-set three-dimensional rendering engine, and the three-dimensional rendering engine is used for rendering output to obtain the target hyper-realistic digital human corresponding to the target face.
[0113] An embodiment of the present invention further provides a device for generating a hyper-realistic digital human, for implementing the method for generating a hyper-realistic digital human described in any of the above embodiments, the device comprising:
[0114] The original face static mesh acquisition module is used to generate the original face static mesh and original facial texture map based on the target face;
[0115] A target face static mesh acquisition module is used to retopologize the original face static mesh according to a preset standard mesh to obtain a target face static mesh;
[0116] A target face facial texture acquisition module is used to generate various textures of the target face static mesh based on the original facial texture map and the target face static mesh;
[0117] A target face skeletal mesh acquisition module is used to transfer the skeleton binding information of the standard mesh to the target face static mesh to generate a target face skeletal mesh;
[0118] The target hyper-realistic digital human acquisition module is used to render the target human face skeleton mesh and the various maps to obtain the target hyper-realistic digital human.
[0119] It should be noted that the device for generating a hyper-realistic digital human provided by an embodiment of the present invention can implement all the processes of the method for generating a hyper-realistic digital human described in any of the above embodiments. The functions and technical effects achieved by each module in the device are respectively the same as those of the method for generating a hyper-realistic digital human described in the above embodiments, and will not be repeated here.
[0120] An embodiment of the present invention further provides a computer-readable storage medium, comprising a stored computer program, which, when run, controls the device where the computer-readable storage medium is located to execute the method for generating a hyper-realistic digital human as described in any of the above embodiments.
[0121] An embodiment of the present invention further provides a computer program product, including a computer program, which, when executed by a processor, implements the method for generating a hyper-realistic digital human as described in any of the above embodiments.
[0122] The embodiment of the present invention also provides a terminal device, see Figure 6 , which is a structural block diagram of a preferred embodiment of a terminal device provided by the present invention, wherein the terminal device includes a processor 10, a memory 20, and a computer program stored in the memory 20 and configured to be executed by the processor 10. When executing the computer program, the processor 10 implements the method for generating a hyper-realistic digital human as described in any of the above embodiments.
[0123] Preferably, the computer program can be divided into one or more modules / units (e.g., computer program 1, computer program 2, ...), which are stored in the memory 20 and executed by the processor 10 to implement the present invention. The one or more modules / units can be a series of computer program instruction segments capable of implementing specific functions, and the instruction segments are used to describe the execution process of the computer program in the terminal device.
[0124] The processor 10 can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor, or the processor 10 can be any conventional processor. The processor 10 is the control center of the terminal device, and uses various interfaces and lines to connect various parts of the terminal device.
[0125] The memory 20 mainly includes a program storage area and a data storage area. The program storage area can store an operating system, at least one application required for a function, and the data storage area can store related data. In addition, the memory 20 can be a high-speed random access memory or a non-volatile memory, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, and a flash card. Alternatively, the memory 20 can be other volatile solid-state memory devices.
[0126] It should be noted that the above terminal device may include, but is not limited to, a processor and a memory. Those skilled in the art will understand that Figure 6 The structural block diagram shown is only an example of the structure of the terminal device and does not constitute a structural limitation of the terminal device. The terminal device may include more or fewer components than shown in the figure, or a combination of certain components, or different components.
[0127] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.
Claims
1. A method for generating a hyper-realistic digital human, characterized in that: include: Generate an original face static mesh and original facial texture map based on the target face; Retopologizing the original face static mesh according to a preset standard mesh to obtain a target face static mesh; Generate various maps of the target face static mesh according to the original facial texture map and the target face static mesh; Transferring the skeleton binding information of the standard mesh to the target face static mesh to generate a target face skeletal mesh; The target human face skeleton mesh and the maps are rendered to obtain a target hyper-realistic digital human.
2. The method for generating a hyper-realistic digital human according to claim 1, wherein: Generating an original face static mesh and an original facial texture map based on the target face specifically includes: When the target face does not participate in the light field scanning, collecting multiple photos corresponding to the target face at multiple angles; The multiple photos are input into a preset 3D reconstruction software, and a static mesh preset in the 3D reconstruction software is used to perform multi-angle feature matching with the multiple photos to perform 3D face reconstruction, and texture coordinate mapping is performed to obtain the original face static mesh and the original facial texture map that matches the original face static mesh.
3. The method for generating a hyper-realistic digital human according to claim 1, wherein: Generating an original face static mesh and an original facial texture map based on the target face specifically includes: When the target face participates in light field scanning, the target face is scanned at multiple angles and multiple positions by a light field scanning device to obtain light information and color information corresponding to the target face; Reconstructing the light information to obtain three-dimensional coordinates of facial feature points of the target face; A texture map is generated according to the color information, and the generated texture map is combined with the three-dimensional coordinates to obtain an original human face static mesh; wherein the original human face static mesh contains the original facial texture map.
4. The method for generating a hyper-realistic digital human according to claim 1, wherein: The retopology of the original human face static mesh according to the preset standard mesh to obtain the target human face static mesh specifically includes: Inputting the preset standard mesh and the original face static mesh into the preset retopology software; The retopology software is used to mark corresponding points on the facial feature points of the two meshes in sequence and then perform wrapping deformation so that the facial feature points of the standard mesh match the facial feature points of the original face static mesh, thereby obtaining the target face static mesh.
5. The method for generating a hyper-realistic digital human according to claim 1, wherein: Generating various maps of the target face static mesh according to the original facial texture map and the target face static mesh specifically includes: Performing normal baking on the target face static mesh, and baking and transferring the original facial texture map to the target face static mesh, thereby obtaining a normal map and a color map; The baked normal map and color map are repaired according to the normal map and color map of the standard mesh, and the repaired normal map and color map are used as the normal map and color map of the target face static mesh.
6. The method for generating a hyper-realistic digital human according to claim 1, wherein: The transferring of the skeleton binding information of the standard mesh to the target face static mesh to generate the target face skeleton mesh specifically includes: Input the standard mesh and the target face static mesh into DCC software; Calculate the bone offset corresponding to each bone node in the target face static mesh according to the bone binding information and vertex coordinates of the standard mesh and the vertex coordinates of the target face static mesh; Move the bone position of each bone node in the target face static mesh according to the bone offset corresponding to each bone node; The target face static mesh is re-bound to the relocated bone nodes, and the bone weights of the standard mesh are transferred to the target face static mesh using the transfer function in the DCC software to generate a target face skeletal mesh.
7. The method for generating a hyper-realistic digital human according to claim 6, wherein: Calculating the bone offset corresponding to each bone node in the target face static mesh according to the bone binding information and vertex coordinates of the standard mesh and the vertex coordinates of the target face static mesh specifically includes: For each bone node in the target face static mesh, determine a target bone node corresponding to the bone node according to the bone binding information of the standard mesh; Determine a first target vertex closest to the target bone node from all vertices of the standard mesh; Determine, based on the first target vertex, a second target vertex corresponding to the first target vertex from all vertices of the target face static mesh; The bone offset corresponding to the bone node is calculated based on the center coordinates of the target bone node, the center coordinates of the first target vertex, and the center coordinates of the second target vertex.
8. A computer-readable storage medium, characterized in that The invention comprises a stored computer program, which controls the device where the computer-readable storage medium is located to execute the method for generating a hyper-realistic digital human according to any one of claims 1 to 7 when the computer program is run.
9. A computer program product, characterized in that The invention comprises a computer program, which, when executed by a processor, implements the method for generating a hyper-realistic digital human according to any one of claims 1 to 7.
10. A terminal device, characterized in that: The invention comprises a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor implements the method for generating a hyper-realistic digital human according to any one of claims 1 to 7 when executing the computer program.