A digital human reconstruction method, device, equipment and storage medium

Abstract

The present disclosure provides a digital human reconstruction method and device, equipment and storage medium, relates to the technical field of artificial intelligence, in particular to the technical field of computer vision, deep learning, large model and the like, and can be applied to the scene of digital human and the like. The specific implementation scheme is: constructing a personalized human body grid based on a human body video or image, the personalized human body grid carrying human body feature information in the human body video or image; extracting a head grid from the personalized human body grid, generating a corresponding head Gaussian representation based on the head grid; generating a corresponding body Gaussian representation based on the personalized human body grid; and fusing the head Gaussian representation and the body Gaussian representation to obtain a full-body Gaussian representation of a digital human. The present disclosure can improve the realism of the human body geometry of the digital human.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, in particular to computer vision, deep learning, large models and other technical fields, and can be applied to scenarios such as digital humans. Background Art

[0002] Photorealistic and drivable 3D human reconstruction is an active research topic in computer vision and graphics. Interactive digital avatars capable of natural body movements and subtle facial expressions offer significant benefits for numerous downstream applications, such as remote presentation, virtual companionship, and augmented reality. However, achieving digital human reconstruction and enhancing the realism of human geometry remain key technical challenges. Summary of the Invention

[0003] The present disclosure provides a digital human reconstruction method, apparatus, device and storage medium.

[0004] According to one aspect of the present disclosure, a digital human reconstruction method is provided, comprising:

[0005] Building a personalized human body mesh based on a human body video or image, the personalized human body mesh carrying human body feature information in the human body video or image;

[0006] Extracting a head mesh from the personalized body mesh, and generating a corresponding head Gaussian representation based on the head mesh;

[0007] Based on the personalized human body mesh, a corresponding body Gaussian representation is generated;

[0008] The head Gaussian representation and the body Gaussian representation are fused to obtain the full-body Gaussian representation of the digital human.

[0009] According to another aspect of the present disclosure, a digital human reconstruction device is provided, comprising:

[0010] A mesh construction module is used to construct a personalized human body mesh based on a human body video or image, where the personalized human body mesh carries human body feature information in the human body video or image;

[0011] A head Gaussian generation module is used to extract a head mesh from the personalized human body mesh and generate a corresponding head Gaussian representation based on the head mesh;

[0012] A body Gaussian generation module, configured to generate a corresponding body Gaussian representation based on the personalized human body mesh;

[0013] The first fusion module is used to fuse the head Gaussian representation and the body Gaussian representation to obtain the whole body Gaussian representation of the digital human.

[0014] According to another aspect of the present disclosure, there is provided an electronic device, comprising:

[0015] at least one processor; and

[0016] a memory communicatively connected to the at least one processor; wherein,

[0017] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform any method in the embodiments of the present disclosure.

[0018] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute any method according to the embodiments of the present disclosure.

[0019] According to another aspect of the present disclosure, a computer program product is provided, including a computer program. When the computer program is executed by a processor, the computer program implements any one of the methods according to the embodiments of the present disclosure.

[0020] The present disclosure performs Gaussian reconstruction of the head and body respectively, and can use different Gaussian representations for the head and body parts, thereby combining the strengths of different Gaussian representations to capture the full body details of the digital human and improve the realism of the digital human's human body geometry.

[0021] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings are provided to facilitate a better understanding of the present invention and do not constitute a limitation of the present disclosure.

[0023] Figure 1 is a flowchart of a digital human reconstruction method according to an embodiment of the present disclosure;

[0024] Figure 2 is a flowchart for implementing reconstruction of a personalized human body grid in one embodiment of the present disclosure;

[0025] Figure 3A is a schematic diagram of an initial human body mesh according to an embodiment of the present disclosure;

[0026] Figure 3B is a schematic diagram of a coarse human body mesh according to an embodiment of the present disclosure;

[0027] Figure 3C is a schematic diagram of an optimized human body mesh according to an embodiment of the present disclosure;

[0028] Figure 3D is a schematic diagram of a personalized human body grid according to an embodiment of the present disclosure;

[0029] Figure 4 is a flowchart for implementing head Gaussian reconstruction in one embodiment of the present disclosure;

[0030] Figure 5 is a flowchart for implementing body Gaussian reconstruction in one embodiment of the present disclosure;

[0031] Figure 6 is a flowchart for implementing Gaussian fusion in one embodiment of the present disclosure;

[0032] Figure 7 is a flowchart for implementing driving a whole-body Gaussian representation in one embodiment of the present disclosure;

[0033] Figure 8 is a framework diagram of a digital human reconstruction method according to an embodiment of the present disclosure;

[0034] Figure 9 is a structural diagram of a digital human reconstruction device 900 according to an embodiment of the present disclosure;

[0035] Figure 10 is a structural diagram of a digital human reconstruction device 1000 according to an embodiment of the present disclosure;

[0036] Figure 11 A schematic block diagram of an example electronic device 1100 is shown, which may be used to implement embodiments of the present disclosure. DETAILED DESCRIPTION

[0037] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be appreciated by those skilled in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0038] The “and / or” in the embodiments of the present disclosure indicates that there may be three relationships. For example, A and / or B may indicate three situations: A exists alone, A and B exist at the same time, and B exists alone. The term “at least one” herein indicates any combination of at least two of any one or more of a plurality of. For example, at least one of A, B, and C may indicate any one or more elements selected from the set consisting of A, B, and C. The terms “first” and “second” herein refer to and distinguish between multiple similar technical terms, and do not mean to limit the order or to limit the meaning to only two. For example, the first feature and the second feature refer to two categories / two features. The first feature may be one or more, and the second feature may also be one or more.

[0039] Photorealistic and drivable 3D human reconstruction has been an active research topic in computer vision and graphics. Interactive digital human avatars can perform natural body movements and subtle facial expressions, bringing significant benefits to numerous downstream applications.

[0040] Early image-based human body modeling methods typically used mesh, point cloud, and signed distance field representations to characterize the geometric shape and texture appearance of the human body. However, due to limited expressiveness, these reconstructed human body geometries are relatively rough and lack high-frequency texture details, which is far from realistic.

[0041] Recently, 3D Gaussian representations have successfully combined photorealism with fast rendering performance, resulting in a rapid emergence of numerous 3D Gaussian virtual image solutions. However, current single Gaussian representations primarily focus on limb reconstruction and are unable to simultaneously accommodate body and head deformation patterns, resulting in poor digital human reconstruction. For example, head expression deformation is primarily based on blend shapes, while body deformation includes joints and non-rigid parts of clothing. The currently widely used mesh-embedded Gaussian representation can handle head deformation but struggles with non-rigid body deformation. Convolutional Neural Network (CNN)-based Gaussian representations can learn non-rigid body deformations but struggle to capture changes in facial expressions.

[0042] To address this situation, an embodiment of the present disclosure proposes a digital human reconstruction method, which implements digital human reconstruction based on combined Gaussian representation. Figure 1 This is a flowchart of a digital human reconstruction method according to an embodiment of the present disclosure, including:

[0043] S110: Constructing a personalized human body mesh based on a human body video or image, where the personalized human body mesh carries human body feature information in the human body video or image;

[0044] S120, extracting a head mesh from the personalized body mesh, and generating a corresponding head Gaussian representation based on the head mesh;

[0045] S130, generating a corresponding Gaussian representation of the body based on the personalized human body mesh;

[0046] S140: Fusing the Gaussian representation of the head and the Gaussian representation of the body to obtain a full-body Gaussian representation of the digital human.

[0047] The above steps S120 and S130 can be executed one after the other or simultaneously; the present disclosure does not limit the execution order of these two steps.

[0048] In one example, a human body video or image may include an image input by a user or a video of a certain length, which includes multiple frames of images; the image or video includes a human body image; using the method proposed in the embodiment of the present disclosure, a full-body Gaussian representation consistent with the human body image can be reconstructed, and the full-body Gaussian representation can be further driven to obtain a dynamic digital human image, which is consistent with the human body image included in the image or video.

[0049] As can be seen from the above, the disclosed embodiment proposes a four-step reconstruction method. First, in step S110, a personalized human network carrying human feature information is constructed based on human videos or images. The personalized human network can be used as a geometric proxy for the digital human. Then, in steps S120 and S130, Gaussian reconstruction of the head and body is performed based on the geometric proxy. Finally, in step S140, Gaussian fusion of the head and body is completed, thereby obtaining a full-body Gaussian representation that can represent the image of the digital human.

[0050] Based on different forms of 3D Gaussian representations, the disclosed embodiments enable full-body, controllable virtual image reconstruction from monocular video. Specifically, because the proposed method performs Gaussian reconstruction of the head and body, respectively, different Gaussian representations can be used for the head and body parts. This leverages the strengths of these different Gaussian representations to capture the full-body details of the digital human, enhancing the digital human's realism.

[0051] In some examples, a mesh-embedded Gaussian representation can be used for head reconstruction to accurately handle head deformation, while a CNN-based Gaussian representation can be used for body reconstruction to accurately handle non-rigid body deformation. Combining the advantages of these two methods, the resulting fused full-body Gaussian representation can accurately handle head deformation and capture changes in facial expressions, while also accurately handling non-rigid body deformation and capturing changes in non-rigid parts such as clothing.

[0052] The following describes the various stages of the above four-step reconstruction method, corresponding to Figure 1 Steps S110 to S140 in:

[0053] Phase 1: Personalized human body mesh reconstruction:

[0054] In one example, at this stage, the input video / image is used to quickly optimize the position of the 3D Gaussian point cloud, and a personalized human body mesh that retains features is extracted as a geometric proxy.

[0055] Figure 2 This is a flowchart of the implementation of reconstructing a personalized human body grid in one embodiment of the present disclosure. Figure 2 As shown, the following steps are included:

[0056] S210, generating a rough human body Gaussian with human body feature information based on a human body video or image;

[0057] S220, according to the rough human body Gaussian, obtain a rough human body grid (referred to as coarse grid, hereinafter referred to as M) carrying the human body feature information. C express);

[0058] S230, based on the rough human body mesh (M C ), for the initial human body mesh (hereinafter referred to as M s ) to obtain the optimized human body mesh (hereinafter referred to as M′ R express);

[0059] S240, based on the optimized human body mesh (M′ R ), and the rough human body mesh (M C ), get the personalized human body mesh (hereinafter using M R express).

[0060] The personalized human body mesh (M R ) can be used as a geometric proxy for the digital human and used in subsequent stages to reconstruct the synchronized Gaussian and body Gaussian.

[0061] In some examples, step S210 includes:

[0062] Constructing the initial human body mesh (M s );

[0063] Based on the initial human body mesh (M s ) Initialize the Gaussian point attributes to obtain the initial human body Gaussian;

[0064] The initial human body Gaussian is optimized based on the human body video or image to obtain a rough human body Gaussian carrying human body feature information.

[0065] In the present example, the initial human mesh (M s ) is a generalized human mesh template that can establish a generalized human Gaussian template, i.e., an initial human Gaussian. The initial human Gaussian does not carry the human information input by the user. Subsequently, due to the optimization of the initial human Gaussian by the user input human video or image, a rough human Gaussian can be obtained, which can carry the human information in the user input human video or image.

[0066] For example, first, the human body is parameterized in the canonical space using an extended skinned multi-person linear (SMPLX) model (which is irrelevant to the input image / video, does not carry the human features in the input human image / video, and does not include clothing features) to initialize Gaussian attributes (such as position, scale, opacity, rotation, spherical harmonic coefficients, etc.) to construct an initial human mesh. Then, using the complete skeleton of the SMPLX model, the Gaussian in the canonical space is deformed to the pose space by linear blending skinning (LBS) or the like. Further, a differentiable rendering is used to obtain an image, which is compared with the real input image / video to calculate the loss, such as the mean-square error (MSE) loss and / or the perceptual loss, between the rendered image and the real input image / video, and the position and other attributes of the Gaussian are optimized based on the loss to obtain a rough human Gaussian.

[0067] In some examples, in step S220, according to the rough human Gaussian, a spherical shell surface reconstruction algorithm can be used to obtain a rough human mesh (M C ) that describes the human feature information (i.e., human shape information).

[0068] At this time, the rough human mesh (M C ) has the problem of missing face and hand details. Therefore, step S230 can be used to optimize the vertex positions of the initial human mesh (M s ) to fit the shape of the rough human mesh (M C ), so that the vertex positions of the initial human mesh (M s ) approximate the vertex positions of the rough human mesh (M C ).

[0069] In some embodiments, during the fitting process of step S230, the structural features of the face and fingers of the optimized initial human mesh (M s ) are preserved. Specifically, step S230 includes:

[0070] The initial human mesh (M s ) is optimized using a three-dimensional semantic mask corresponding to a human hand and / or face, and the optimization excludes optimization of the human hand and / or face to obtain an optimized human mesh (M' R ).

[0071] In particular, to preserve the structural features, a fine mesh is obtained. An embodiment of the present disclosure aligns a parameterized SMPLX initial human mesh (M s ) with a reconstructed coarse human mesh (M C ) in a regular space, and optimizes the vertex positions of the SMPLX initial human mesh (M s ) to fit the shape of the coarse human mesh (M C ), wherein, during the optimization, the vertices of the SMPLX initial human mesh (M s ) are approximated to the vertex positions of the coarse human mesh (M C ), and the optimized human mesh is denoted as M' R .

[0072] In an example, to preserve the structural features of the face and fingers, the hand and face vertices of the initial human mesh (M s ) do not participate in the shape optimization, for example, using a 3D semantic mask to control the vertices of the hand and / or face of the initial human mesh (M s ) not participating in the optimization, so as to include the structural features of the hand and / or face in the initial human mesh (M s ).

[0073] The above optimization process can be represented by equation (1):

[0074] L mesh = |S h,f M′ R ―S h,f M C | 2 +γ reg L lap (M′ R ) (1)

[0075] wherein,

[0076] M′ R denotes the optimized human mesh;

[0077] M C denotes the coarse human mesh;

[0078] S h,f denotes a three-dimensional semantic mask corresponding to a human hand and face, which is used to mask M' R and MC Multiplication can exclude the optimization of the vertices of the human hand and face during the optimization process;

[0079] |*| 2 Indicates calculating the difference between corresponding vertices in two meshes;

[0080] L lap (M′ R ) represents the Laplace regularization term, which is used to maintain the optimized human body mesh M′ R Avoid local distortion to constrain the optimized human body mesh M′ R The shape is smooth and reasonable;

[0081] γ reg Indicates the weight of the Laplace regularization term, which can be set to an empirical value, such as 0.1;

[0082] L mesh Represents the loss function; in the optimization process, by adjusting the initial human body mesh (M s ) vertex positions, so that the adjusted mesh (i.e. M′ R ) vertex approximation to the rough human body mesh (M C ) vertex position; therefore, in the optimization process, L mesh tends to smaller values.

[0083] When the number of adjustments reaches the preset threshold, the initial human body mesh (M s ) is optimized and the final optimized human body mesh (M′) is obtained. R ).

[0084] Optimized human body mesh (M′ R ) carries the human feature information in the input human video / image and retains the detailed features of the hands and face; however, the optimized human mesh (M′ R ) is different from the topological structure of the real human body wearing shoes, which will interfere with the result. In order to solve this problem, the embodiment of the present disclosure adopts the above step S240 to transform the rough human body mesh (M C )The foot shape is replaced with the optimized human body mesh (M′ R ) to obtain a personalized human body mesh (M R ), that is, personalized human body mesh (M R ) including the coarse human body mesh (M C ) of the human foot area and the optimized human mesh (M′ R ) refers to the area other than the human foot.

[0085] This process can be expressed as follows:

[0086] M R =Concate(M′ R | vbody ,M C | vfoot ) (2)

[0087] in,

[0088] vbody represents the vertices of the body in the mesh;

[0089] vfoot represents the vertex of the foot in the mesh;

[0090] Concate(*) means connecting mesh vertices.

[0091] like Figures 3A-3D They are respectively the initial human body mesh (M) of an embodiment of the present disclosure. s ), coarse human body mesh (M C ), optimized human body mesh (M′ R ), personalized human body grid (M R ) schematic diagram.

[0092] The second stage, head Gaussian reconstruction:

[0093] In this example, in this stage, the personalized human body mesh (M R ) and embed the new 3D Gaussian into the local coordinate system of the face, and use the pre-established multi-expression head dataset for training to achieve 3D head Gaussian reconstruction.

[0094] Considering that the facial deformation mode is mainly controlled by the blend shape, the embodiment of the present disclosure can adopt a mesh-embedded three-dimensional Gaussian representation to restrict the 3D Gaussian to the local space, which can make full use of the head mesh prior to obtain a realistic three-dimensional Gaussian result of the head with controllable expression and head posture.

[0095] Figure 4 is a flowchart of the implementation of reconstructing the head Gaussian in one embodiment of the present disclosure, such as Figure 4 As shown, the following steps are included:

[0096] S410, using the head mesh, generating an initial head Gaussian;

[0097] S420. Optimize the initial head Gaussian using a pre-established multi-expression head dataset to obtain a head Gaussian representation; wherein the multi-expression head dataset includes: multiple head images corresponding to different expressions, and expression parameters corresponding to each head image.

[0098] In the above process, since only the head mesh in the personalized human body mesh is used to generate the head Gaussian representation, that is, the Gaussian representation is generated only for the head, a different Gaussian representation method can be used for the head than for other body parts, which makes it easier to take advantage of various different Gaussian representation methods.

[0099] In one example, a predefined mapping is used to generate a personalized human body mesh (M R ) to extract the personalized Flame head mesh (head mesh for short).

[0100] In one example, step S410 includes:

[0101] For each triangle in the head mesh, assign a Gaussian point to the triangle and set the initial local properties of the Gaussian point;

[0102] Determine the initial global properties of the Gaussian point based on the initial local properties of the Gaussian point and the global properties of the triangle;

[0103] An initial head Gaussian is generated based on the initial global properties of each Gaussian point in the head mesh.

[0104] In one example, in step S420, the process of optimizing the initial head Gaussian includes:

[0105] Drive the initial head Gaussian based on the expression parameters to generate the target head Gaussian corresponding to the expression parameters;

[0106] Based on the head image corresponding to the expression parameter, adjusting the target head Gaussian so that the rendered image of the target head Gaussian approaches the head image corresponding to the expression parameter;

[0107] Based on the adjusted target head Gaussian, a head Gaussian representation is determined.

[0108] As can be seen from the above process, during the process of determining the head Gaussian representation, the initial head Gaussian is optimized using a pre-established multi-expression head dataset. The disclosed embodiments can pre-establish a multi-expression head dataset, which includes multiple head images with different expressions and expression parameters corresponding to each head image. During the optimization of the initial head Gaussian, the initial head Gaussian is iteratively adjusted multiple times based on each "head image-expression parameter" pair in the multi-expression head dataset. For example, the initial head Gaussian is driven based on the expression parameters, a corresponding rendered image is obtained based on the driven head Gaussian, and the difference between the rendered image and the head image corresponding to the expression parameters is determined; the head Gaussian is adjusted based on the difference; then, the adjusted head Gaussian is again driven based on the expression parameters, the difference between the rendered image and the head image corresponding to the expression parameters is determined, and the head Gaussian is adjusted based on the difference; this iterative adjustment is repeated until the rendered image approaches the head image corresponding to the expression parameters, ultimately obtaining an adjusted target head Gaussian, which is then used as the head Gaussian representation.

[0109] For example, for each triangle in the head mesh, a local coordinate system can be constructed and Gaussian points assigned. Each triangle can be assigned attributes, including a local coordinate origin, a rotation matrix, and a scaling factor. For example, the mean position t can be used as the local coordinate origin of the triangle; the rotation matrix r can be defined based on the triangle's edge direction vectors, normal vectors, and their cross product vectors; and the scalar K can be introduced to describe the triangle's scaling factor.

[0110] For Gaussian points embedded in a triangle, the initial local properties of the Gaussian points can be initialized. The initial local properties may include: local position μ, rotation matrix R, and scaling factor S. For example, the local position μ is initialized to the local origin, the rotation matrix r is initialized to the identity matrix, and the scaling factor s is initialized to the unit vector. During rendering, the local properties of the Gaussian points can be converted to global properties in the global space. For example, the following formula (3) is used to determine the global properties:

[0111] r′=Rr,μ′=kRμ+T,s′=ks (3)

[0112] Among them, r′ is the global rotation matrix of the Gaussian point, μ′ is the global origin of the Gaussian point, and s′ is the global scaling factor of the Gaussian point.

[0113] In one example, during rendering, the blend shape vectors, head bones, and skin weights provided by the Flame model are used to drive the global head Gaussian (i.e., the head Gaussian represented by global attributes) to achieve changes in expression and head posture.

[0114] The third stage, body Gaussian reconstruction:

[0115] In one example, at this stage, a pose map (PoseMap) is constructed based on the personalized human body mesh, and CNN (such as the StyleUNet network) is combined to predict the Gaussian attribute map related to the pose, thereby realizing three-dimensional body Gaussian reconstruction.

[0116] The main challenge in human Gaussian reconstruction is how to model non-rigid clothing deformations that depend on posture. To improve generalization, the disclosed embodiments introduce a CNN architecture (such as the StyleUnet network) to predict body pose-dependent Gaussians, thereby simulating clothing deformations.

[0117] Figure 5 is a flowchart of the implementation of body Gaussian reconstruction in one embodiment of the present disclosure, such as Figure 5 As shown, the following steps are included:

[0118] S510, based on personalized human body grid (M R ), generate basic human body Gaussian;

[0119] S520, constructing a human body posture map (PoseMap) using the basic human body Gaussian;

[0120] S530: Input the posture graph into a convolutional neural network (CNN) model to be optimized to obtain a Gaussian attribute residual output by the CNN model;

[0121] S540, optimizing the basic human Gaussian and the CNN model based on a pre-established multi-pose body dataset and the Gaussian attribute residual;

[0122] S550, generating a body Gaussian representation based on the optimized basic human Gaussian and convolutional neural network model;

[0123] The pre-established multi-posture body dataset includes: a plurality of body images corresponding to different postures, and body parameters corresponding to each body image.

[0124] In the above process, a Gaussian representation method different from that of the head can be used for body parts, which makes it easier to give full play to the advantages of various different Gaussian representation methods.

[0125] In one example, S540 includes:

[0126] Add the Gaussian attribute residual to the basic human body Gaussian to obtain the added human body Gaussian;

[0127] Performing posture deformation on the added human body Gaussian using body parameters to obtain a posture-deformed Gaussian;

[0128] Based on the body image corresponding to the body parameters, the basic human Gaussian and convolutional neural network model are optimized to adjust the Gaussian after the posture deformation, so that the rendered image of the Gaussian after the posture deformation approaches the body image corresponding to the body parameters.

[0129] In some embodiments, a personalized body mesh (M R ) initializes the basic human body Gaussian and constructs the front and back PoseMaps, which are then input into a CNN model (such as a StyleUNet network), which outputs a Gaussian attribute residual used to describe non-rigid deformations. Further, the Gaussian residual is added to the aforementioned basic human body Gaussian, and the Gaussian points are deformed based on LBS to obtain Gaussian points in different postures (determined by body parameters). Finally, based on the Gaussian renderings of the Gaussian points in different postures and the body images corresponding to the body parameters, loss functions such as reconstruction loss, perception loss, and residual rule loss are calculated between the two, thereby optimizing the StyleUnet network to learn the correct Gaussian non-rigid deformation and optimize the basic human body Gaussian. Afterwards, based on the optimized StyleUnet network and the basic human body Gaussian, a body Gaussian representation is generated. Using a convolutional neural network model (such as a StyleUnet network) to generate a full-body Gaussian representation of a digital human, that is, using a Gaussian expression based on a CNN network to reconstruct the body Gaussian, can improve the generation efficiency and accuracy of the full-body Gaussian representation.

[0130] It can be seen from the above process that in the process of determining the Gaussian representation of the body, the pre-established multi-pose body data set is used to optimize the basic human Gaussian and StyleUnet networks. The disclosed embodiment can pre-establish a multi-pose body data set, which includes multiple body images in different poses, and body parameters corresponding to each body image (such as pose parameters and action parameters). In the process of optimizing the basic human Gaussian and StyleUnet networks, based on each "body image-body parameter" pair in the multi-pose body data set, the basic human Gaussian and StyleUnet networks are iteratively adjusted multiple times to make the Gaussian rendering image approach the body image corresponding to the body parameters, and finally obtain the adjusted basic human Gaussian and StyleUnet networks; then, based on the adjusted basic human Gaussian and StyleUnet networks, a body Gaussian representation is generated.

[0131] The fourth stage, Gaussian fusion:

[0132] In the second and third stages above, Gaussian representations of the head and body were generated, respectively. In this stage, the Gaussian representations of the head generated in the second stage and the Gaussian representations of the body generated in the third stage are aligned and fused.

[0133] In one example, the personalized human body mesh (M R ) is used as a guide to complete the Gaussian alignment, splicing and fusion of the head and body, and finally achieve seamless joint rendering.

[0134] Figure 6 This is a flowchart of Gaussian fusion implementation in one embodiment of the present disclosure. Figure 6 As shown, the following steps are included:

[0135] S610: Remove the lower portion of the midline of the Gaussian representation of the head (generated in the second stage) to obtain the Gaussian content of the head; and remove the upper portion of the midline of the Gaussian representation of the body (generated in the third stage) to obtain the Gaussian content of the body.

[0136] S620: Combine the head Gaussian content and the body Gaussian content to obtain a full-body Gaussian representation of the digital human.

[0137] In one example, the "centerline" represents the stitching boundary at the neck of a person in the Gaussian representation. Removing the lower portion of the centerline from the head Gaussian representation retains only the portion above the neck. Removing the upper portion of the centerline from the body Gaussian representation retains only the portion below the neck. Combining these retained representations yields a full-body Gaussian representation of the person. This full-body Gaussian representation combines the advantages of the mesh-embedded 3D Gaussian representation of the head and the CNN-predicted Gaussian representation of the body. It combines the strengths of both Gaussian representations while being compatible with deformation patterns of both the head and body.

[0138] Furthermore, because the head and body Gaussians come from different Gaussian representations, the head and body can be decoupled and driven separately during joint driving and rendering. Specifically, the head Gaussian is responsible for driving expressions, while the body Gaussian is responsible for driving the entire body. This separate driving approach improves the accuracy and flexibility of driving the digital human.

[0139] Figure 7 is a flowchart of driving the whole-body Gaussian representation in one embodiment of the present disclosure, such as Figure 7 As shown, the following steps are included:

[0140] S710, acquiring expression parameters and whole body posture parameters in the driving signal;

[0141] S720: Drive the head Gaussian in the whole-body Gaussian representation based on the expression parameter in the drive signal; and drive the body Gaussian in the whole-body Gaussian representation based on the whole-body posture parameter in the drive signal;

[0142] S730: Fusing the driven head Gaussian and body Gaussian to obtain a driving result corresponding to the driving signal.

[0143] Among them, the whole body posture parameters may include parameters such as body shape and movement.

[0144] In one example, for a given driving signal (including expression parameters and whole-body posture parameters), the expression parameters are first used to perform a mixed deformation on the head model (that is, the head Gaussian representation) to obtain a head Gaussian with expression offset. Then, the posture parameters are used to construct a posture graph, and the posture graph is input into StyleUNet to obtain a residual Gaussian. The residual Gaussian is then added to the basic human body Gaussian to obtain a standardized body Gaussian. Furthermore, the head Gaussian and the body Gaussian are aligned and fused to obtain a whole-body Gaussian. Finally, the skeleton of the SMPLX model is used to deform the entire body Gaussian to the target posture and render it to obtain the driving result.

[0145] The above introduces the four-step reconstruction method for realizing digital human reconstruction in the embodiment of the present disclosure. In the four-step reconstruction process, a pre-established multi-expression head dataset and a multi-pose body dataset are used to establish the head Gaussian and body Gaussian. Due to the limited expressiveness of the parameterized human body model, the current full-body motion capture algorithm cannot accurately estimate the limbs, fingers and facial expressions at the same time, resulting in the established dataset being inaccurate. However, accurate motion annotation (i.e., constructing an accurate dataset) is the basis for reconstructing digital humans from images. Based on this, the digital human reconstruction method proposed in the embodiment of the present disclosure further includes a preprocessing stage, which can respectively establish a multi-expression head dataset and a multi-pose body dataset to improve the accuracy of establishing the Gaussian representation in the subsequent four-step reconstruction process.

[0146] Figure 8 is a framework diagram of a digital human reconstruction method according to an embodiment of the present disclosure, such as Figure 8 As shown, the digital human reconstruction method proposed in the embodiment of the present disclosure includes a preprocessing stage 810 and a four-step reconstruction stage 820; wherein, the preprocessing stage 810 can establish a multi-expression head dataset and a multi-pose body dataset; the subsequent four-step reconstruction stage 820 performs Gaussian generation based on the multi-expression head dataset and the multi-pose body dataset.

[0147] In some embodiments, the process of establishing a multi-expression head dataset includes:

[0148] Based on a human body video or image, multiple head images are determined, each head image corresponding to a different expression;

[0149] Based on the facial detail capture algorithm, each head image is calculated to obtain the expression parameters corresponding to the head image;

[0150] A multi-expression head dataset is constructed based on multiple head images and the expression parameters corresponding to each head image.

[0151] In some embodiments, the process of creating a multi-pose body dataset includes:

[0152] Based on human body videos or images, multiple body images, each body image corresponds to a different posture;

[0153] Based on the human motion capture algorithm and the hand motion capture algorithm, each body image is calculated to obtain the body parameters corresponding to the body image;

[0154] A multi-pose body dataset is constructed based on multiple body images and body parameters corresponding to each body image.

[0155] Reconstructing a full-body, controllable 3D Gaussian figure from an input human image or video requires accurate estimation of facial pose and expression, hand pose, and body pose, as well as good registration of image features. Current mainstream all-inclusive full-body pose estimation models struggle to accurately achieve satisfactory pose results and often exhibit significant misalignment in areas such as fingers, facial features, and toes.

[0156] To solve this problem, the present invention designs a multi-estimator joint method in the preprocessing stage. The core of the method is to improve the accuracy of the joint whole-body posture by utilizing multiple optimal posture estimators for body parts, and establish a multi-expression head dataset and a multi-pose body dataset respectively.

[0157] For example, given a monocular video image, for the head, the Detailed Expression Capture and Animation (DECA) algorithm is used, combined with facial keypoint optimization, to estimate expression parameters (such as head pose, shape, and expression) based on the Flame representation of the head. For the body, a full-body pose estimator, such as the Pyramidal Mesh Alignment Feedback-X (PyMAF-X) algorithm, is first used to obtain preliminary body parameters (such as body pose parameters and shape parameters) based on the SMPLX representation of the whole body. At this time, the hands are usually misaligned; therefore, a Hand Mesh Recovery (HaMER) estimator can be further introduced to estimate the hand pose, shape, and hand keypoints based on the Metric-Affine Hand Model (MANO) representation of the hand. The hand keypoints are then used to further optimize the body pose parameters, resulting in optimized body parameters with well-aligned hand joints.

[0158] Based on the optimized data, we can construct a multi-expression head dataset and a multi-pose body dataset for training the head Gaussian and body Gaussian, respectively. The multi-expression head dataset contains multiple head images corresponding to different expressions and the expression parameters corresponding to each head image, while the multi-pose body dataset contains multiple body images corresponding to different poses and the body parameters corresponding to each body image.

[0159] The present disclosure also provides a digital human reconstruction device. Figure 9 FIG. 1 is a structural diagram of a digital human reconstruction device 900 according to an embodiment of the present disclosure, comprising:

[0160] A mesh construction module 910 is configured to construct a personalized human mesh based on a human video or image, wherein the personalized human mesh carries human feature information in the human video or image;

[0161] A head Gaussian generation module 920 is configured to extract a head mesh from the personalized body mesh and generate a corresponding head Gaussian representation based on the head mesh;

[0162] A body Gaussian generation module 930 is configured to generate a corresponding body Gaussian representation based on the personalized human body mesh;

[0163] The first fusion module 940 is used to fuse the head Gaussian representation and the body Gaussian representation to obtain the whole body Gaussian representation of the digital human.

[0164] In some embodiments, the grid construction module 910 is configured to:

[0165] Based on a human body video or image, a rough human body Gaussian with the human body feature information is generated;

[0166] According to the rough human body Gaussian, a rough human body mesh carrying the characteristic information of the human body is obtained;

[0167] Based on the rough human body mesh, the initial human body mesh is optimized to obtain an optimized human body mesh;

[0168] The personalized human body mesh is obtained based on the optimized human body mesh and the rough human body mesh.

[0169] In some embodiments, the grid construction module 910 is configured to:

[0170] Construct the initial human body mesh;

[0171] Initializing Gaussian point attributes based on the initial human body mesh to obtain an initial human body Gaussian;

[0172] optimizing the initial human body Gaussian based on human body videos or images to obtain a coarse human body Gaussian carrying human body feature information.

[0173] In some embodiments, the mesh constructing module 910 is configured to:

[0174] In some embodiments, the mesh constructing module 910 is configured to:

[0175] In some embodiments, the mesh constructing module 910 is configured to:

[0176] In some embodiments, the mesh constructing module 910 is configured to:

[0177] In some embodiments, the head Gaussian generating module 920 is configured to:

[0178] In some embodiments, the head Gaussian generating module 920 is configured to:

[0179] In some embodiments, the head Gaussian generating module 920 is configured to:

[0180] In some embodiments, the head Gaussian generating module 920 is configured to:

[0181] In some embodiments, the head Gaussian generating module 920 is configured to:

[0182] In some embodiments, the head Gaussian generating module 920 is configured to:

[0183] In some embodiments, the head Gaussian generating module 920 is configured to:

[0184] In some embodiments, the head Gaussian generating module 920 is configured to:

[0185] In some embodiments, the head Gaussian generating module 920 is configured to:

[0186] In some embodiments, the head Gaussian generating module 920 is configured to:

[0187] Based on the adjusted target head Gaussian, a head Gaussian representation is determined.

[0188] In some embodiments, the body Gaussian generation module 930 is configured to:

[0189] Generate basic human Gaussian based on personalized human mesh;

[0190] Using the basic human body Gaussian, a human body posture graph is constructed;

[0191] The posture graph is input into the convolutional neural network model to be optimized to obtain the Gaussian attribute residual output by the convolutional neural network model;

[0192] Based on the pre-established multi-pose body dataset and Gaussian attribute residuals, the basic human Gaussian and convolutional neural network models are optimized;

[0193] Generate a Gaussian representation of the body based on the optimized basic human Gaussian and convolutional neural network models;

[0194] The multi-posture body dataset includes: a plurality of body images corresponding to different postures, and body parameters corresponding to each body image.

[0195] In some embodiments, the body Gaussian generation module 930 is configured to:

[0196] Add the Gaussian attribute residual to the basic human body Gaussian to obtain the added human body Gaussian;

[0197] Performing posture deformation on the added human body Gaussian using body parameters to obtain a posture-deformed Gaussian;

[0198] Based on the body image corresponding to the body parameters, the basic human Gaussian and convolutional neural network models are optimized to adjust the Gaussian after the posture deformation, so that the rendered image of the Gaussian after the posture deformation approaches the body image corresponding to the body parameters.

[0199] In some embodiments, the first fusion module 940 is configured to:

[0200] The upper portion of the midline of the Gaussian representation of the head is removed to obtain the Gaussian content of the head; and the upper portion of the midline of the Gaussian representation of the body is removed to obtain the Gaussian content of the body;

[0201] The head Gaussian content and the body Gaussian content are combined to obtain a full-body Gaussian representation of the digital human.

[0202] Figure 10 FIG. 1 is a schematic structural diagram of a digital human reconstruction apparatus 1000 according to an embodiment of the present disclosure. In some implementations, the apparatus further includes:

[0203] An acquisition module 1050 is used to acquire expression parameters and whole body posture parameters in the driving signal;

[0204] A driving module 1060 is configured to drive the head Gaussian in the whole-body Gaussian representation based on the expression parameter in the driving signal; and to drive the body Gaussian in the whole-body Gaussian representation based on the whole-body posture parameter in the driving signal;

[0205] The second fusion module 1070 is used to fuse the driven head Gaussian and body Gaussian to obtain a driving result corresponding to the driving signal.

[0206] In some embodiments, the digital human reconstruction device 1000 further includes:

[0207] A head data set establishment module 1080 is used to determine a plurality of head images based on a human body video or image, each head image corresponding to a different expression;

[0208] Based on the facial detail capture algorithm, each head image is calculated to obtain the expression parameters corresponding to the head image;

[0209] A multi-expression head dataset is constructed based on the multiple head images and expression parameters corresponding to each head image.

[0210] In some embodiments, the digital human reconstruction device 1000 further includes:

[0211] A body data set establishment module 1090 is configured to determine a plurality of body images based on a human body video or image, each body image corresponding to a different posture;

[0212] Based on the human motion capture algorithm and the hand motion capture algorithm, each body image is calculated to obtain the body parameters corresponding to the body image;

[0213] A multi-pose body dataset is constructed based on multiple body images and body parameters corresponding to each body image.

[0214] For the description of specific functions and examples of each module and submodule of the device in the embodiment of the present disclosure, please refer to the relevant description of the corresponding steps in the above method embodiment, which will not be repeated here.

[0215] In the technical solution disclosed herein, the acquisition, storage and application of personal information of users involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0216] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0217] Figure 11A schematic block diagram of an example electronic device 1100 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are provided as examples only and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0218] like Figure 11 As shown, the device 1100 includes a computing unit 1101, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1102 or a computer program loaded from a storage unit 1108 into a random access memory (RAM) 1103. Various programs and data required for the operation of the device 1100 can also be stored in the RAM 1103. The computing unit 1101, the ROM 1102, and the RAM 1103 are connected to each other via a bus 1104. An input / output (I / O) interface 1105 is also connected to the bus 1104.

[0219] Various components in device 1100 are connected to I / O interface 1105, including: an input unit 1106, such as a keyboard, mouse, etc.; an output unit 1107, such as various types of displays, speakers, etc.; a storage unit 1108, such as a magnetic disk, optical disk, etc.; and a communication unit 1109, such as a network card, modem, wireless communication transceiver, etc. The communication unit 1109 allows device 1100 to exchange data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0220] The computing unit 1101 can be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1101 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1101 performs the various methods and processes described above, such as the detection method. For example, in some embodiments, the detection method can be implemented as a computer software program that is tangibly contained in a machine-readable medium, such as a storage unit 1108. In some embodiments, part or all of the computer program can be loaded and / or installed on the device 1100 via the ROM 1102 and / or the communication unit 1109. When the computer program is loaded into the RAM 1103 and executed by the computing unit 1101, one or more steps of the detection method described above can be performed. Alternatively, in other embodiments, the computing unit 1101 can be configured to perform the detection method in any other appropriate manner (e.g., by means of firmware).

[0221] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system comprising at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0222] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0223] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0224] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0225] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0226] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact through a communication network. The client-server relationship arises through computer programs running on the respective computers and having a client-server relationship with each other. The server may be a cloud server, a server in a distributed system, or a server integrated with a blockchain.

[0227] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not limited herein.

[0228] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. A digital human reconstruction method, comprising: Building a personalized human body mesh based on a human body video or image, wherein the personalized human body mesh carries human body feature information in the human body video or image; Extracting a head mesh from the personalized body mesh, and generating an initial head Gaussian using the head mesh; Optimizing the initial head Gaussian using a pre-established multi-expression head dataset to obtain a head Gaussian representation; The multi-expression head data set includes a plurality of head images corresponding to different expressions, and expression parameters corresponding to each of the head images; generating a basic human body Gaussian based on the personalized human body mesh; A posture graph of the human body is constructed using the basic human body Gaussian; the posture graph is input into a convolutional neural network model to be optimized, and the convolutional neural network model outputs Gaussian attribute residuals; the basic human body Gaussian and the convolutional neural network model are optimized based on a pre-established multi-posture body dataset and the Gaussian attribute residuals; a body Gaussian representation is generated based on the optimized basic human body Gaussian and the convolutional neural network model; the multi-posture body dataset includes multiple body images corresponding to different postures, and body parameters corresponding to each of the body images; The head Gaussian representation and the body Gaussian representation are fused to obtain the full-body Gaussian representation of the digital human.

2. The method according to claim 1, wherein The process of constructing a personalized human body mesh based on a human body video or image includes: Based on the human body video or image, generating a rough human body Gaussian that carries the human body feature information; Obtaining a rough human body mesh carrying the human body feature information according to the rough human body Gaussian; Based on the rough human body mesh, optimizing the initial human body mesh to obtain an optimized human body mesh; The personalized human body mesh is obtained based on the optimized human body mesh and the rough human body mesh.

3. The method according to claim 2, wherein: Generating a rough human body Gaussian carrying the human body feature information based on the human body video or image includes: Constructing the initial human body mesh; Initializing Gaussian point attributes based on the initial human body mesh to obtain an initial human body Gaussian; The initial human body Gaussian is optimized based on the human body video or image to obtain a rough human body Gaussian carrying the human body feature information.

4. The method according to claim 2 or 3, wherein: The optimizing the initial human body mesh based on the rough human body mesh includes: The initial human body mesh is optimized using a three-dimensional semantic mask corresponding to the human body hands and / or face, and optimization of the human body hands and / or face is excluded during the optimization process.

5. The method according to claim 2 or 3, wherein: The obtaining of the personalized human body mesh based on the optimized human body mesh and the rough human body mesh includes: The human foot region of the coarse human body mesh is replaced with the foot region of the optimized human body mesh to obtain the personalized human body mesh; wherein the personalized human body mesh includes the human foot region of the coarse human body mesh and the region other than the human foot of the optimized human body mesh.

6. The method according to claim 1, wherein The generating an initial head Gaussian using the head mesh includes: For each triangle in the head mesh, assign a Gaussian point to the triangle and set initial local attributes of the Gaussian point; Determining initial global properties of the Gaussian points based on the initial local properties of the Gaussian points and the properties of the triangle; An initial head Gaussian is generated based on the initial global properties of each Gaussian point in the head grid.

7. The method according to claim 1, wherein The optimizing the initial head Gaussian using a pre-established multi-expression head dataset to obtain the head Gaussian representation includes: driving the initial head Gaussian based on the expression parameter to generate a target head Gaussian corresponding to the expression parameter; Based on the head image corresponding to the expression parameters, adjusting the target head Gaussian so that the rendered image of the target head Gaussian approaches the head image corresponding to the expression parameters; The head Gaussian representation is determined based on the adjusted target head Gaussian.

8. The method according to claim 1, wherein The method of optimizing the basic human Gaussian and the convolutional neural network model based on the pre-established multi-pose body dataset and the Gaussian attribute residual includes: Adding the Gaussian attribute residual to the basic human body Gaussian to obtain an added human body Gaussian; Performing posture deformation on the added human body Gaussian using the body parameters to obtain a posture-deformed Gaussian; Based on the body image corresponding to the body parameters, the basic human Gaussian and the convolutional neural network model are optimized to adjust the posture-deformed Gaussian so that the rendered image of the posture-deformed Gaussian approaches the body image corresponding to the body parameters.

9. The method according to claim 1, wherein The fusing of the head Gaussian representation and the body Gaussian representation to obtain the whole-body Gaussian representation of the digital human includes: removing the lower portion of the midline of the Gaussian representation of the head to obtain the Gaussian content of the head; and removing the upper portion of the midline of the Gaussian representation of the body to obtain the Gaussian content of the body; The head Gaussian content and the body Gaussian content are combined to obtain a full-body Gaussian representation of the digital human.

10. The method according to claim 1, further comprising, Obtain expression parameters and whole body posture parameters in the driving signal; driving the head Gaussian in the whole-body Gaussian representation based on the expression parameter in the driving signal; and driving the body Gaussian in the whole-body Gaussian representation based on the whole-body posture parameter in the driving signal; The driven head Gaussian and body Gaussian are fused to obtain a driving result corresponding to the driving signal.

11. The method according to claim 7, further comprising: Determining a plurality of head images based on the human body video or image, each of the head images corresponding to a different expression; Based on a facial detail capture algorithm, calculating each of the head images respectively to obtain expression parameters corresponding to the head images; The multi-expression head dataset is constructed based on the multiple head images and the expression parameters corresponding to each of the head images.

12. The method according to claim 1 or 8, further comprising: Determining a plurality of body images based on the human body video or image, each of the body images corresponding to a different posture; Based on a human motion capture algorithm and a hand motion capture algorithm, respectively calculating each of the body images to obtain body parameters corresponding to the body image; The multi-posture body dataset is constructed based on the multiple body images and the body parameters corresponding to each of the body images.

13. A digital human reconstruction device, comprising: A grid construction module, configured to construct a personalized human body grid based on a human body video or image, wherein the personalized human body grid carries human body feature information in the human body video or image; A head Gaussian generation module is used to extract a head mesh from the personalized human body mesh and generate an initial head Gaussian using the head mesh; Optimizing the initial head Gaussian using a pre-established multi-expression head dataset to obtain a head Gaussian representation; The multi-expression head data set includes a plurality of head images corresponding to different expressions, and expression parameters corresponding to each of the head images; A body Gaussian generation module, configured to generate a basic human body Gaussian based on the personalized human body mesh; A posture graph of the human body is constructed using the basic human body Gaussian; the posture graph is input into a convolutional neural network model to be optimized, and the convolutional neural network model outputs Gaussian attribute residuals; the basic human body Gaussian and the convolutional neural network model are optimized based on a pre-established multi-posture body dataset and the Gaussian attribute residuals; a body Gaussian representation is generated based on the optimized basic human body Gaussian and the convolutional neural network model; the multi-posture body dataset includes multiple body images corresponding to different postures, and body parameters corresponding to each of the body images; The first fusion module is used to fuse the head Gaussian representation and the body Gaussian representation to obtain the whole body Gaussian representation of the digital human.

14. The device according to claim 13, wherein The grid construction module is used to: Based on the human body video or image, generating a rough human body Gaussian that carries the human body feature information; Obtaining a rough human body mesh carrying the human body feature information according to the rough human body Gaussian; Based on the rough human body mesh, optimizing the initial human body mesh to obtain an optimized human body mesh; The personalized human body mesh is obtained based on the optimized human body mesh and the rough human body mesh.

15. The device according to claim 13, wherein The grid construction module is used to: Constructing the initial human body mesh; Initializing Gaussian point attributes based on the initial human body mesh to obtain an initial human body Gaussian; The initial human body Gaussian is optimized based on the human body video or image to obtain a rough human body Gaussian carrying the human body feature information.

16. The device according to claim 14 or 15, wherein The grid construction module is used to: The initial human body mesh is optimized using a three-dimensional semantic mask corresponding to the human body hands and / or face, and optimization of the human body hands and / or face is excluded during the optimization process.

17. The device according to claim 14 or 15, wherein The grid construction module is used to: The human foot region of the coarse human body mesh is replaced with the foot region of the optimized human body mesh to obtain the personalized human body mesh; wherein the personalized human body mesh includes the human foot region of the coarse human body mesh and the region other than the human foot of the optimized human body mesh.

18. The device according to claim 13, wherein The head Gaussian generation module is used for: For each triangle in the head mesh, assign a Gaussian point to the triangle and set initial local attributes of the Gaussian point; Determining initial global properties of the Gaussian points based on the initial local properties of the Gaussian points and the properties of the triangle; An initial head Gaussian is generated based on the initial global properties of each Gaussian point in the head grid.

19. The device according to claim 13, wherein The head Gaussian generation module is used for: driving the initial head Gaussian based on the expression parameter to generate a target head Gaussian corresponding to the expression parameter; Based on the head image corresponding to the expression parameters, adjusting the target head Gaussian so that the rendered image of the target head Gaussian approaches the head image corresponding to the expression parameters; The head Gaussian representation is determined based on the adjusted target head Gaussian.

20. The apparatus according to claim 13, wherein The body Gaussian generation module is used to: Adding the Gaussian attribute residual to the basic human body Gaussian to obtain an added human body Gaussian; Performing posture deformation on the added human body Gaussian using the body parameters to obtain a posture-deformed Gaussian; Based on the body image corresponding to the body parameters, the basic human Gaussian and the convolutional neural network model are optimized to adjust the posture-deformed Gaussian so that the rendered image of the posture-deformed Gaussian approaches the body image corresponding to the body parameters.

21. The apparatus according to claim 13, wherein The first fusion module is configured to: removing the lower portion of the midline of the Gaussian representation of the head to obtain the Gaussian content of the head; and removing the upper portion of the midline of the Gaussian representation of the body to obtain the Gaussian content of the body; The head Gaussian content and the body Gaussian content are combined to obtain a full-body Gaussian representation of the digital human.

22. The apparatus according to claim 13, further comprising: An acquisition module, used to obtain expression parameters and whole body posture parameters in the driving signal; a driving module, configured to drive the head Gaussian in the whole-body Gaussian representation based on the expression parameter in the driving signal; and driving a body Gaussian in the whole-body Gaussian representation based on the whole-body posture parameter in the driving signal; The second fusion module is used to fuse the driven head Gaussian and body Gaussian to obtain a driving result corresponding to the driving signal.

23. The apparatus of claim 19, further comprising: A head data set establishment module, configured to determine a plurality of head images based on the human body video or image, each of the head images corresponding to a different expression; Based on a facial detail capture algorithm, calculating each of the head images respectively to obtain expression parameters corresponding to the head images; The multi-expression head dataset is constructed based on the multiple head images and the expression parameters corresponding to each of the head images.

24. The apparatus according to claim 13 or 20, further comprising: a body data set establishing module, configured to determine a plurality of body images based on the human body video or image, each of the body images corresponding to a different posture; Based on the human motion capture algorithm and the hand motion capture algorithm, each of the body images is calculated to obtain the body parameters corresponding to the body image; based on the multiple body images and the body parameters corresponding to each of the body images, the multi-posture body dataset is constructed.

25. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 12.

26. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-12.

27. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 12.

Images