Digital human reconstruction method, apparatus and device, and storage medium

Abstract

The invention provides a digital human reconstruction method, device and equipment and a storage medium, relates to the technical field of artificial intelligence, in particular to the technical fields of computer vision, deep learning, large models and the like, and can be applied to scenes such as digital human. According to the specific implementation scheme, a personalized human body grid is constructed based on a human body video or image, wherein the personalized human body grid carries human body feature information in the human body video or image; extracting a head grid from the personalized human body grid, and generating a corresponding head Gaussian representation based on the head grid; generating a corresponding body Gaussian representation based on the personalized human body grid; and fusing the head Gaussian representation and the body Gaussian representation to obtain a whole body Gaussian representation of the digital human. The reality sense of the geometric shape of the human body of the digital human can be improved.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, in particular to technical fields such as computer vision, deep learning, and large models, and can be applied to scenarios such as digital humans. Background Art

[0002] Photorealistic and drivable 3D human reconstruction is an active research topic in computer vision and graphics. Interactive digital images can make natural body movements and subtle facial expressions, which can bring significant benefits to many downstream applications, such as remote display, virtual companionship and augmented reality. How to achieve digital human reconstruction and improve the realism of digital human body geometry is a technical problem that needs to be solved. Summary of the invention

[0003] The present disclosure provides a digital human reconstruction method, device, equipment and storage medium.

[0004] According to one aspect of the present disclosure, a digital human reconstruction method is provided, comprising:

[0005] Building a personalized human body mesh based on a human body video or image, the personalized human body mesh carrying human body feature information in the human body video or image;

[0006] Extracting a head mesh from the personalized human body mesh, and generating a corresponding head Gaussian representation based on the head mesh;

[0007] Based on the personalized human body mesh, a corresponding body Gaussian representation is generated;

[0008] The head Gaussian representation and the body Gaussian representation are fused to obtain the full body Gaussian representation of the digital human.

[0009] According to another aspect of the present disclosure, there is provided a digital human reconstruction device, comprising:

[0010] A mesh construction module, used to construct a personalized human mesh based on a human video or image, the personalized human mesh carrying human feature information in the human video or image;

[0011] A head Gaussian generation module is used to extract a head mesh from the personalized human body mesh, and generate a corresponding head Gaussian representation based on the head mesh;

[0012] A body Gaussian generation module, used for generating a corresponding body Gaussian representation based on the personalized human body mesh;

[0013] The first fusion module is used to fuse the head Gaussian representation and the body Gaussian representation to obtain the whole body Gaussian representation of the digital human.

[0014] According to another aspect of the present disclosure, there is provided an electronic device, comprising:

[0015] at least one processor; and

[0016] a memory communicatively connected to the at least one processor; wherein,

[0017] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute any method in the embodiments of the present disclosure.

[0018] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute any method according to the embodiments of the present disclosure.

[0019] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, which implements any method according to the embodiments of the present disclosure when executed by a processor.

[0020] The present disclosure performs Gaussian reconstruction of the head and body respectively, and can use different Gaussian representations for the head and body parts, thereby combining the strengths of different Gaussian representations to capture the full body details of the digital human and improve the realism of the digital human's human body geometry.

[0021] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0023] Figure 1 is a flow chart of a method for reconstructing a digital human according to an embodiment of the present disclosure;

[0024] Figure 2 is a flowchart for implementing reconstruction of a personalized human body grid in one embodiment of the present disclosure;

[0025] Figure 3A is a schematic diagram of an initial human body mesh according to an embodiment of the present disclosure;

[0026] Figure 3B is a schematic diagram of a coarse human body mesh according to an embodiment of the present disclosure;

[0027] Figure 3C is a schematic diagram of an optimized human body mesh according to an embodiment of the present disclosure;

[0028] Figure 3D is a schematic diagram of a personalized human body grid according to an embodiment of the present disclosure;

[0029] Figure 4 is a flowchart of implementing the reconstruction of head Gaussian in one embodiment of the present disclosure;

[0030] Figure 5 is a flowchart of implementing body Gaussian reconstruction in one embodiment of the present disclosure;

[0031] Figure 6 is a flowchart for implementing Gaussian fusion in one embodiment of the present disclosure;

[0032] Figure 7 is a flowchart for implementing driving the whole-body Gaussian representation in one embodiment of the present disclosure;

[0033] Figure 8 is a framework diagram of a digital human reconstruction method according to an embodiment of the present disclosure;

[0034] Fig. 9 is a structural schematic diagram of a digital human reconstruction device 900 according to an embodiment of the present disclosure;

[0035] Fig.10 is a structural schematic diagram of a digital human reconstruction device 1000 according to an embodiment of the present disclosure;

[0036] Fig.11 A schematic block diagram of an example electronic device 1100 is shown that can be used to implement embodiments of the present disclosure. DETAILED DESCRIPTION

[0037] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0038] The “and / or” in the embodiments of the present disclosure indicates that three relationships may exist. For example, A and / or B may indicate the following three situations: A exists alone, A and B exist at the same time, and B exists alone. The term “at least one” herein indicates any combination of at least two of any one or more of a plurality of types. For example, at least one of A, B, and C may indicate any one or more elements selected from the set consisting of A, B, and C. The terms “first” and “second” herein refer to and distinguish between multiple similar technical terms, and do not mean to limit the order or to limit the meaning to only two. For example, the first feature and the second feature refer to two types / two features. The first feature may be one or more, and the second feature may also be one or more.

[0039] Photorealistic and drivable 3D human reconstruction has been an active research topic in computer vision and graphics. Interactive digital human avatars can make natural body movements and subtle facial expressions, which can bring significant benefits to many downstream applications.

[0040] Early image-based human body modeling methods usually used mesh, point cloud and signed distance field representations to characterize the geometric shape and texture appearance of the human body. However, due to limited expressiveness, these reconstructed human body geometries are relatively rough and lack high-frequency texture details, which is far from realistic.

[0041] Recently, 3D Gaussian has successfully combined photorealism with fast rendering performance, so many 3D Gaussian virtual image solutions have emerged in a short period of time. However, the current single Gaussian representation mainly focuses on limb reconstruction and cannot be compatible with the deformation modes of the body and head at the same time, resulting in poor digital human reconstruction. For example, the head expression deformation is mainly based on blend shapes, and the body deformation includes joints and non-rigid parts of clothing; the currently widely used mesh embedded Gaussian representation can handle head deformation but has difficulty in handling non-rigid body deformation; the Gaussian representation based on convolutional neural networks (CNN) can learn non-rigid body deformation but has difficulty capturing facial expression changes.

[0042] In view of this situation, an embodiment of the present disclosure proposes a digital human reconstruction method, which realizes digital human reconstruction based on combined Gaussian representation. Figure 1 is a flowchart of a method for reconstructing a digital human according to an embodiment of the present disclosure, comprising:

[0043] S110, constructing a personalized human body grid based on a human body video or image, where the personalized human body grid carries human body feature information in the human body video or image;

[0044] S120, extracting a head mesh from the personalized human body mesh, and generating a corresponding head Gaussian representation based on the head mesh;

[0045] S130, generating a corresponding Gaussian representation of the body based on the personalized human body mesh;

[0046] S140: Fusing the head Gaussian representation and the body Gaussian representation to obtain a full-body Gaussian representation of the digital human.

[0047] The above steps S120 and S130 may be executed one after the other or simultaneously; the present disclosure does not limit the execution order of these two steps.

[0048] In one example, a human body video or image may include an image input by a user or a video of a certain length, which includes multiple frames of images; the image or video includes a human body image; using the method proposed in the embodiment of the present disclosure, a full-body Gaussian representation consistent with the human body image can be reconstructed, and the full-body Gaussian representation can be further driven to obtain a dynamic digital human image, which is consistent with the human body image included in the image or video.

[0049] It can be seen from the above content that the embodiment of the present disclosure proposes a four-step reconstruction method. First, step S110 is used to construct a personalized human network carrying human feature information based on human videos or images, and the personalized human network can be used as a geometric proxy for the digital human; then, steps S120 and S130 are respectively used to perform Gaussian reconstruction of the head and Gaussian reconstruction of the body based on the geometric proxy; finally, step S140 is used to complete the Gaussian fusion of the head and the body, thereby obtaining a full-body Gaussian representation that can represent the image of the digital human.

[0050] Based on different forms of 3D Gaussian representations, the embodiments of the present disclosure can achieve full-body controllable virtual image reconstruction from monocular video. Specifically, since the method proposed in the present disclosure performs head Gaussian reconstruction and body Gaussian reconstruction on the head and body parts respectively, different Gaussian representations can be used for the head and body parts, thereby combining the strengths of different Gaussian representations to capture the full-body details of the digital human and improve the realism of the digital human.

[0051] In some examples, the head Gaussian reconstruction can be performed using the mesh embedded Gaussian expression to accurately handle the head deformation, and the body Gaussian reconstruction can be performed using the CNN network-based Gaussian expression to accurately handle the body non-rigid deformation. Combining the advantages of the above two, the whole body Gaussian representation obtained after fusion can not only accurately handle the head deformation and capture the changes in facial expressions, but also accurately handle the non-rigid deformation of the body and capture the changes in non-rigid parts such as clothes.

[0052] The following describes each stage of the above four-step reconstruction method, corresponding to Figure 1 Steps S110 to S140 in:

[0053] Phase 1: Personalized human body mesh reconstruction:

[0054] In one example, at this stage, the input video / image is used to quickly optimize the position of the 3D Gaussian point cloud, and a personalized human body mesh that retains the features is extracted as a geometric proxy.

[0055] Figure 2 is a flowchart of the implementation of reconstructing a personalized human body grid in one embodiment of the present disclosure, such as Figure 2 As shown, the following steps are included:

[0056] S210, generating a rough human body Gaussian carrying human body feature information based on a human body video or image;

[0057] S220, according to the rough human body Gaussian, a rough human body mesh (referred to as the rough mesh, hereinafter referred to as M) carrying the human body feature information is obtained. C express);

[0058] S230, based on the rough human body mesh (M C ), for the initial human body mesh (hereinafter referred to as M s denoted by) to obtain the optimized human body mesh (hereinafter referred to as M′ R express);

[0059] S240, based on the optimized human body mesh (M′ R ), and the rough human body mesh (M C ), and obtain the personalized human body mesh (hereinafter, M R express).

[0060] The personalized human body mesh (M R ) can be used as a geometric proxy for the digital human and used in the subsequent stages to reconstruct the synchronized Gaussian and body Gaussian.

[0061] In some examples, step S210 includes:

[0062] Constructing the initial human body mesh (M s );

[0063] Based on the initial human body mesh (M s ) Initialize the Gaussian point attributes to obtain the initial human body Gaussian;

[0064] The initial human body Gaussian is optimized based on the human body video or image to obtain a rough human body Gaussian carrying human body feature information.

[0065] In this example, the initial human mesh (M s ) is a generalized human body mesh template, which can establish a generalized human body Gaussian template, namely, an initial human body Gaussian; the initial human body Gaussian does not carry the human body information input by the user, and then, since the initial human body Gaussian is optimized using the human body video or image input by the user, an optimized rough human body Gaussian can be obtained, and the rough human body Gaussian can carry the human body information in the human body video or image input by the user.

[0066] For example, the human body is first parameterized in the canonical space using the extended Skinned Multi-Person Linear Model-X (SMPLX) model (the human body is independent of the input image / video, does not carry the character features in the input human body image / video, and does not include clothing features) to initialize Gaussian attributes (such as position, scale, opacity, rotation, spherical harmonic coefficients, etc.), thereby constructing an initial human body mesh. Then, using the complete skeleton of the SMPLX model, the Gaussian in the canonical space is deformed to the pose space by linear blending skinning (LBS) and other methods. Further, differentiable rendering is used to obtain an image and compare it with the real input image / video to calculate the loss between the rendered image and the real input image / video, such as mean square error (MSE) loss and / or perceptual loss, and then the position and other attributes of the Gaussian are optimized based on the loss to obtain a rough human body Gaussian.

[0067] In some examples, in step S220, a rough human body mesh (M) describing human body feature information (i.e., human body shape information) can be obtained by using a spherical shell surface reconstruction algorithm based on the rough human body Gaussian. C ).

[0068] At this time, the rough human body mesh (M C ) has the problem of losing facial and hand details. Therefore, step S230 can be adopted to optimize the initial human body mesh (M s ) to fit the rough human body mesh (M C ) shape, so that the initial human body mesh (M s ) vertex position approximates the rough human body mesh (M C )’s vertex position.

[0069] In some implementations, during the fitting process of step S230, the optimized initial human body mesh (M s )'s face and finger structural features. Specifically, step S230 includes:

[0070] The initial human mesh (M s ) is optimized, and the optimization of the human hand and / or face is excluded during the optimization process to obtain the optimized human mesh (M′ R ).

[0071] Specifically, in order to retain the structural features and obtain a fine mesh, an embodiment of the present disclosure aligns the parameterized SMPLX initial human body mesh (M s ) and the reconstructed rough human body mesh (M C ), and optimize the SMPLX initial human mesh (M s ) to fit the rough human body mesh (M C ) shape, during the optimization process, the SMPLX initial human body mesh (M s ) vertex approximation to the rough human body mesh (M C ), the optimized human body mesh is recorded as M′ R .

[0072] In one example, in order to preserve the structural features of the face and fingers, the initial human body mesh (M s ) do not participate in shape optimization, for example, shape optimization is performed using a 3D semantic mask that is used to control the initial human mesh (M s )'s hands and / or face vertices are not included in the optimization, thus including the initial human mesh (M s ) Structural features of the hands and / or face.

[0073] The above optimization process can be expressed by formula (1):

[0074] L mesh =|S h,f M′ R ―S h,f M C | 2 +γ reg L lap (M′ R ) (1)

[0075] in,

[0076] M′ R represents the optimized human body mesh;

[0077] M C represents a rough human body mesh;

[0078] S h,f Represents the 3D semantic mask corresponding to the human hand and face, and the 3D semantic mask is used to identify M′ R and MC Multiplication can exclude the optimization of the vertices of the human hand and face during the optimization process;

[0079] |*| 2 It means calculating the difference between corresponding vertices in two meshes;

[0080] L lap (M′ R ) represents the Laplace regularization term, which is used to maintain the optimized human body mesh M′ R Do not have local distortion problems to constrain the optimized human body mesh M′ R The shape is smooth and reasonable;

[0081] γ reg Indicates the weight of the Laplace regularization term, which can be set to an empirical value, such as 0.1;

[0082] L mesh Represents the loss function; in the optimization process, by adjusting the initial human body mesh (M s ) vertex positions, so that the adjusted mesh (i.e., M′ R ) vertex approximation to the rough human body mesh (M C ) vertex position; therefore, in the optimization process, L mesh tends to smaller values.

[0083] When the number of adjustments reaches a preset threshold, the initial human body mesh (M s ) is optimized, and the final optimized human body mesh (M′) is obtained. R ).

[0084] Optimized human body mesh (M′ R ) not only carries the human feature information in the input human video / image, but also retains the detailed features of the hands and face; however, the optimized human mesh (M′ R ) is different from the topological structure of the real human body wearing shoes, which will interfere with the result. In order to solve this problem, the embodiment of the present disclosure adopts the above step S240 to transform the rough human body mesh (M C )The shape of the foot is replaced with the optimized human body mesh (M′ R ) to obtain a personalized human body mesh (M R ), that is, personalized human body mesh (M R ) including the coarse human body mesh (M C ) of the human foot area and the optimized human mesh (M′ R ) except the human foot.

[0085] This process can be expressed by the following formula (2):

[0086] M R =Concate(M′ R | vbody ,M C | vfoot ) (2)

[0087] in,

[0088] vbody represents the vertices of the body in the mesh;

[0089] vfoot represents the vertex of the foot in the mesh;

[0090] Concate(*) means connecting mesh vertices.

[0091] like Figures 3A-3D are respectively an initial human body mesh (M) of an embodiment of the present disclosure. s ), coarse human body mesh (M C ), optimized human body mesh (M′ R ), personalized human mesh (M R ) schematic diagram.

[0092] The second stage, head Gaussian reconstruction:

[0093] In this example, in this stage, the personalized human body mesh (M R ) and embed the new 3D Gaussian into the local coordinate system of the face, and train it using a pre-established multi-expression head dataset to achieve 3D head Gaussian reconstruction.

[0094] Considering that the facial deformation mode is mainly controlled by the blend shape, the embodiment of the present disclosure can adopt a mesh-embedded three-dimensional Gaussian representation to restrict the 3D Gaussian to the local space, which can make full use of the head mesh prior to obtain a realistic three-dimensional Gaussian result of the head with controllable expression and head posture.

[0095] Figure 4 is a flowchart of the implementation of reconstructing the head Gaussian in one embodiment of the present disclosure, such as Figure 4 As shown, the following steps are included:

[0096] S410, using the head mesh, generating an initial head Gaussian;

[0097] S420. Utilizing a pre-established multi-expression head data set, the initial head Gaussian is optimized to obtain a head Gaussian representation; wherein the multi-expression head data set includes: a plurality of head images corresponding to different expressions, and expression parameters corresponding to each head image.

[0098] In the above process, since only the head mesh in the personalized human body mesh is used to generate the head Gaussian representation, that is, the Gaussian representation is generated only for the head, a Gaussian representation different from that of the body parts can be used for the head, which makes it easier to give full play to the advantages of various different Gaussian representations.

[0099] In one example, a predefined mapping is used to obtain a personalized human body mesh (M R ) to extract the personalized Flame head mesh (referred to as head mesh) from the Flame Head Grid Editor.

[0100] In one example, step S410 includes:

[0101] For each triangle in the head mesh, a Gaussian point is assigned to the triangle and the initial local attributes of the Gaussian point are set;

[0102] Determine the initial global property of the Gauss point based on the initial local property of the Gauss point and the global property of the triangle;

[0103] Based on the initial global properties of each Gaussian point in the head mesh, an initial head Gaussian is generated.

[0104] In one example, in step S420, the process of optimizing the initial head Gaussian includes:

[0105] Driving the initial head Gaussian based on the expression parameters to generate a target head Gaussian corresponding to the expression parameters;

[0106] Based on the head image corresponding to the expression parameter, the target head Gaussian is adjusted so that the rendered image of the target head Gaussian approaches the head image corresponding to the expression parameter;

[0107] Based on the adjusted target head Gaussian, a head Gaussian representation is determined.

[0108] It can be seen from the above process that in the process of determining the head Gaussian representation, the initial head Gaussian is optimized using the pre-established multi-expression head data set. The disclosed embodiment can pre-establish a multi-expression head data set, which includes multiple head images of different expressions and expression parameters corresponding to each head image. In the process of optimizing the initial head Gaussian, the initial head Gaussian is iteratively adjusted multiple times based on each "head image-expression parameter" pair in the multi-expression head data set. For example, the initial head Gaussian is driven based on the expression parameter, and the corresponding rendering image is obtained according to the driven head Gaussian, and the difference between the rendering image and the head image corresponding to the expression parameter is determined; the head Gaussian is adjusted based on the difference; then, the adjusted head Gaussian is driven again based on the expression parameter, and the difference between the rendering image and the head image corresponding to the expression parameter is determined, and the head Gaussian is adjusted based on the difference; the iterative adjustment is repeated so that the rendering image approaches the head image corresponding to the expression parameter, and finally the adjusted target head Gaussian is obtained, and the adjusted target head Gaussian is used as the head Gaussian representation.

[0109] For example, for each triangle of the head mesh, a local coordinate system can be constructed and Gaussian points can be assigned. For each triangle, attributes can be assigned to the triangle, including the local coordinate origin, rotation matrix, and scaling factor. For example, the average position t is used as the local coordinate origin of the triangle; the rotation matrix r of the triangle is defined based on the edge direction vector, normal vector, and cross product vector of the triangle; and the scalar K is introduced to describe the scaling factor of the triangle.

[0110] For Gaussian points embedded in a triangle, initial local properties of the Gaussian points can be initialized, and the initial local properties may include: local position μ, rotation matrix R, and scaling factor S. For example, the local position μ is initialized as the local origin, the rotation matrix r is initialized as the unit matrix, and the scaling factor s is initialized as the unit vector. During rendering, the local properties of the Gaussian points can be converted into global properties in the global space. For example, the following formula (3) is used to determine the global properties:

[0111] r′=Rr,μ′=kRμ+T,s′=ks (3)

[0112] Among them, r′ is the global rotation matrix of the Gaussian point, μ′ is the global origin of the Gaussian point, and s′ is the global scaling factor of the Gaussian point.

[0113] In one example, during rendering, the blend shape vectors, head bones, and skin weights provided by the Flame model are used to drive the global head Gaussian (i.e., the head Gaussian represented by global attributes) to achieve changes in expression and head posture.

[0114] The third stage, body Gaussian reconstruction:

[0115] In one example, at this stage, a pose map (PoseMap) is constructed based on the personalized human body mesh, and CNN (such as StyleUNet network) is combined to predict the Gaussian attribute map related to the pose, thereby realizing three-dimensional body Gaussian reconstruction.

[0116] The main challenge of human body Gaussian reconstruction is how to model the non-rigid clothing deformation that depends on posture. To improve the generalization ability, the disclosed embodiment introduces a CNN architecture (such as StyleUnet network) to predict the body Gaussian that depends on the posture of the body parts, thereby simulating clothing deformation.

[0117] Figure 5 is a flowchart of the implementation of reconstructing the Gaussian of the body in one embodiment of the present disclosure, such as Figure 5 As shown, the following steps are included:

[0118] S510, based on personalized human body grid (M R ), generate basic human body Gaussian;

[0119] S520, constructing a human body posture map (PoseMap) using the basic human body Gaussian;

[0120] S530, inputting the posture graph into a convolutional neural network (CNN) model to be optimized to obtain a Gaussian attribute residual output by the CNN model;

[0121] S540, optimizing the basic human body Gaussian and the CNN model based on the pre-established multi-posture body data set and the Gaussian attribute residual;

[0122] S550, generating a body Gaussian representation based on the optimized basic human Gaussian and convolutional neural network model;

[0123] The pre-established multi-posture body data set includes: a plurality of body images corresponding to different postures, and body parameters corresponding to each body image.

[0124] In the above process, a Gaussian representation method different from that of the head can be used for body parts, which makes it easier to give full play to the advantages of various different Gaussian representation methods.

[0125] In one example, S540 includes:

[0126] Add the Gaussian attribute residual to the basic human body Gaussian to obtain the added human body Gaussian;

[0127] Using body parameters to perform posture deformation on the added human body Gaussian to obtain a posture-deformed Gaussian;

[0128] Based on the body image corresponding to the body parameters, the basic human Gaussian and convolutional neural network model are optimized to adjust the Gaussian after the posture deformation, so that the rendered image of the Gaussian after the posture deformation approaches the body image corresponding to the body parameters.

[0129] In some embodiments, a personalized human body mesh (M R ) Initialize the basic human Gaussian, and construct the front and back PoseMaps, then input the front and back PoseMaps into the CNN model (such as the StyleUNet network), and the CNN model outputs the Gaussian attribute residual used to describe the non-rigid deformation. Further, the Gaussian residual is added to the aforementioned basic human Gaussian, and the Gaussian points are deformed based on the LBS, so as to obtain Gaussian points in different postures (determined by body parameters). Finally, based on the Gaussian renderings of the Gaussian points in different postures and the body images corresponding to the body parameters, the reconstruction loss, perception loss, residual rule loss and other loss functions between the two are calculated, so as to optimize the StyleUnet network to learn the correct Gaussian non-rigid deformation and optimize the basic human Gaussian. Afterwards, based on the optimized StyleUnet network and the basic human Gaussian, the body Gaussian representation is generated. Using a convolutional neural network model (such as the StyleUnet network) to generate a full-body Gaussian representation of a digital human, that is, using a Gaussian expression based on a CNN network to reconstruct the body Gaussian, the generation efficiency and accuracy of the full-body Gaussian representation can be improved.

[0130] It can be seen from the above process that in the process of determining the Gaussian representation of the body, the pre-established multi-pose body data set is used to optimize the basic human Gaussian and StyleUnet networks. The disclosed embodiment can pre-establish a multi-pose body data set, which includes multiple body images in different poses, and body parameters (such as pose parameters and action parameters) corresponding to each body image. In the process of optimizing the basic human Gaussian and StyleUnet networks, based on each "body image-body parameter" pair in the multi-pose body data set, the basic human Gaussian and StyleUnet networks are iteratively adjusted multiple times to make the Gaussian rendering image approach the body image corresponding to the body parameters, and finally obtain the adjusted basic human Gaussian and StyleUnet networks; then based on the adjusted basic human Gaussian and StyleUnet networks, the body Gaussian representation is generated.

[0131] The fourth stage, Gaussian fusion:

[0132] In the second and third stages above, Gaussian representations of the head and body are generated respectively. In this stage, the Gaussian representation of the head generated in the second stage and the Gaussian representation of the body generated in the third stage are aligned and fused.

[0133] In one example, the personalized human body mesh (M R ) is used as a guide to complete the Gaussian alignment, splicing and fusion of the head and body, and finally achieve seamless joint rendering.

[0134] Figure 6 is a flowchart of Gaussian fusion in one embodiment of the present disclosure, such as Figure 6 As shown, the following steps are included:

[0135] S610, removing the lower part of the midline of the Gaussian representation of the head (generated in the second stage) to obtain the Gaussian content of the head; and removing the upper part of the midline of the Gaussian representation of the body (generated in the third stage) to obtain the Gaussian content of the body;

[0136] S620: Combine the head Gaussian content with the body Gaussian content to obtain a full-body Gaussian representation of the digital human.

[0137] In one example, the "midline" above represents the splicing boundary at the neck of the human body in the Gaussian representation. By removing the lower part of the midline of the Gaussian representation of the head, only the part above the neck in the Gaussian representation of the head is retained; by removing the upper part of the midline of the Gaussian representation of the body, only the part below the neck in the Gaussian representation of the body is retained; by combining the aforementioned retained contents, the whole-body Gaussian representation corresponding to the human body can be obtained. The whole-body Gaussian representation includes the three-dimensional Gaussian representation of the head using grid embedding and the Gaussian representation of the body predicted using the CNN architecture. Therefore, it can combine the advantages of the aforementioned two Gaussian representations and be compatible with the deformation modes of the body and the head.

[0138] Furthermore, since the head Gaussian and the body Gaussian come from different Gaussian representations, the head and the body can be decoupled and driven separately during joint driving and rendering. Specifically, the head Gaussian is responsible for driving the expression, and the body Gaussian is responsible for driving the entire body. The separate driving method can improve the accuracy and flexibility of driving the digital human.

[0139] Figure 7 is a flowchart of driving the whole body Gaussian representation in one embodiment of the present disclosure, such as Figure 7 As shown, the following steps are included:

[0140] S710, acquiring expression parameters and whole body posture parameters in the driving signal;

[0141] S720, driving the head Gaussian in the whole body Gaussian representation based on the expression parameter in the driving signal; and driving the body Gaussian in the whole body Gaussian representation based on the whole body posture parameter in the driving signal;

[0142] S730: Fusing the driven head Gaussian and body Gaussian to obtain a driving result corresponding to the driving signal.

[0143] Among them, the whole body posture parameters may include parameters such as body shape and movement.

[0144] In one example, for a given driving signal (including expression parameters and whole body posture parameters), the head model (that is, the head Gaussian representation) is firstly blended and deformed using the expression parameters to obtain the head Gaussian with expression offset. Then, a posture graph is constructed using the posture parameters, and the posture graph is input into StyleUNet to obtain the residual Gaussian, and then the residual Gaussian is added to the basic human body Gaussian to obtain the standardized body Gaussian. Furthermore, the head Gaussian and the body Gaussian are aligned and fused to obtain the whole body Gaussian. Finally, the entire body Gaussian is deformed to the target posture using the skeleton of the SMPLX model and rendered to obtain the driving result.

[0145] The above introduces the four-step reconstruction method for realizing digital human reconstruction in the embodiment of the present disclosure. In the four-step reconstruction process, a pre-established multi-expression head dataset and a multi-posture body dataset are used to establish the head Gaussian and the body Gaussian. Due to the limited expressiveness of the parameterized human body model, the current full-body motion capture algorithm cannot accurately estimate the limbs, fingers and facial expressions at the same time, resulting in the established dataset being inaccurate. However, accurate motion annotation (i.e., constructing an accurate dataset) is the basis for reconstructing digital humans from images. Based on this, the digital human reconstruction method proposed in the embodiment of the present disclosure further includes a preprocessing stage, which can respectively establish a multi-expression head dataset and a multi-posture body dataset to improve the accuracy of establishing the Gaussian representation in the subsequent four-step reconstruction process.

[0146] Figure 8 is a framework diagram of a digital human reconstruction method according to an embodiment of the present disclosure, such as Figure 8 As shown, the digital human reconstruction method proposed in the embodiment of the present disclosure includes a preprocessing stage 810 and a four-step reconstruction stage 820; wherein, in the preprocessing stage 810, a multi-expression head dataset and a multi-posture body dataset can be established; and in the subsequent four-step reconstruction stage 820, Gaussian generation is performed based on the multi-expression head dataset and the multi-posture body dataset.

[0147] In some embodiments, the process of establishing a multi-expression head dataset includes:

[0148] Based on a human body video or image, a plurality of head images are determined, each head image corresponding to a different expression;

[0149] Based on the facial detail capture algorithm, each head image is calculated to obtain the expression parameters corresponding to the head image;

[0150] Based on multiple head images and the expression parameters corresponding to each head image, a multi-expression head dataset is constructed.

[0151] In some embodiments, the process of establishing a multi-pose body dataset includes:

[0152] Based on human body videos or images, multiple body images, each body image corresponds to a different posture;

[0153] Based on the human motion capture algorithm and the hand motion capture algorithm, each body image is calculated respectively to obtain the body parameters corresponding to the body image;

[0154] A multi-pose body dataset is constructed based on multiple body images and body parameters corresponding to each body image.

[0155] Reconstructing a full-body controllable 3D Gaussian person based on the input human image / video requires accurate estimation of facial pose and expression, hand pose, and body pose, and good registration of image features. The current mainstream all-inclusive full-body pose estimation model is difficult to accurately obtain satisfactory pose results, and usually has obvious misalignment in areas such as fingers, facial features, and toes.

[0156] In order to solve this problem, the present invention designs a multi-estimator joint method in the preprocessing stage. The core of the method is to improve the accuracy of the joint whole-body posture by utilizing multiple optimal posture estimators for body parts, and to establish a multi-expression head dataset and a multi-posture body dataset respectively.

[0157] For example, given a monocular video image, for the head, the Detailed Expression Capture and Animation (DECA) algorithm is used in combination with the optimization of facial key points to estimate the expression parameters (such as head posture, shape and expression, etc.) based on the Flame representation for the head. For the body, a full-body posture estimator is first used, such as the Pyramidal Mesh Alignment Feedback-X (PyMAF-X) algorithm, based on the SMPLX representation for the whole body, to obtain preliminary body parameters (such as body posture parameters and shape parameters, etc.), at which time the hands are usually misaligned; therefore, a Hand Mesh Recovery (HaMER) estimator can be further introduced to estimate the hand posture, shape and hand key points based on the Metric-Affine Hand Model (MANO) representation for the hand, and then the hand key points are used to further optimize the body posture parameters, thereby obtaining optimized body parameters with good hand joint alignment.

[0158] Based on the above optimized data, a multi-expression head dataset and a multi-pose body dataset can be constructed for training head Gaussian and body Gaussian respectively. The multi-expression head dataset contains multiple head images corresponding to different expressions and the expression parameters corresponding to each head image, and the multi-pose body dataset contains multiple body images corresponding to different poses and the body parameters corresponding to each body image.

[0159] The disclosed embodiment also provides a digital human reconstruction device. Fig. 9 is a structural diagram of a digital human reconstruction device 900 according to an embodiment of the present disclosure, comprising:

[0160] A grid construction module 910 is used to construct a personalized human body grid based on a human body video or image, and the personalized human body grid carries human body feature information in the human body video or image;

[0161] A head Gaussian generation module 920 is used to extract a head mesh from the personalized human body mesh, and generate a corresponding head Gaussian representation based on the head mesh;

[0162] A body Gaussian generation module 930 is used to generate a corresponding body Gaussian representation based on the personalized human body mesh;

[0163] The first fusion module 940 is used to fuse the head Gaussian representation and the body Gaussian representation to obtain the whole body Gaussian representation of the digital human.

[0164] In some implementations, the grid construction module 910 is used to:

[0165] Based on a human body video or image, a rough human body Gaussian carrying the characteristic information of the human body is generated;

[0166] According to the rough human body Gaussian, a rough human body mesh carrying characteristic information of the human body is obtained;

[0167] Based on the rough human body mesh, the initial human body mesh is optimized to obtain an optimized human body mesh;

[0168] The personalized human body mesh is obtained based on the optimized human body mesh and the rough human body mesh.

[0169] In some implementations, the grid construction module 910 is used to:

[0170] Construct the initial human body mesh;

[0171] Initializing Gaussian point attributes based on the initial human body mesh to obtain an initial human body Gaussian;

[0172] The initial human body Gaussian is optimized based on the human body video or image to obtain a rough human body Gaussian carrying human body feature information.

[0173] In some implementations, the grid construction module 910 is used to:

[0174] The initial human body mesh is optimized using a three-dimensional semantic mask corresponding to the human body hands and / or face, and the optimization of the human body hands and / or face is excluded during the optimization process.

[0175] In some implementations, the grid construction module 910 is used to:

[0176] The human foot region of the rough human body mesh is replaced with the foot region of the optimized human body mesh to obtain a personalized human body mesh; wherein the personalized human body mesh includes the human foot region of the rough human body mesh and the region other than the human foot in the optimized human body mesh.

[0177] In some embodiments, the head Gaussian generation module 920 is used to:

[0178] Using the head mesh, generate the initial head Gaussian;

[0179] The initial head Gaussian is optimized using a pre-established multi-expression head data set to obtain a head Gaussian representation; wherein the multi-expression head data set includes: a plurality of head images corresponding to different expressions, and expression parameters corresponding to each head image.

[0180] In some embodiments, the head Gaussian generation module 920 is used to:

[0181] For each triangle in the head mesh, assign Gaussian points to the triangle and set the initial local properties of the Gaussian points;

[0182] Determine the initial global properties of the Gaussian points based on the initial local properties of the Gaussian points and the properties of the triangles;

[0183] Based on the initial global properties of each Gaussian point in the head mesh, an initial head Gaussian is generated.

[0184] In some embodiments, the head Gaussian generation module 920 is used to:

[0185] Driving the initial head Gaussian based on the expression parameters to generate a target head Gaussian corresponding to the expression parameters;

[0186] Based on the head image corresponding to the expression parameter, adjusting the target head Gaussian so that the rendered image of the target head Gaussian approaches the head image corresponding to the expression parameter;

[0187] Based on the adjusted target head Gaussian, a head Gaussian representation is determined.

[0188] In some embodiments, the body Gaussian generation module 930 is used to:

[0189] Generate basic human Gaussian based on personalized human mesh;

[0190] Using the basic human body Gaussian, a human body posture graph is constructed;

[0191] The posture graph is input into the convolutional neural network model to be optimized to obtain the Gaussian attribute residual output by the convolutional neural network model;

[0192] Based on the pre-established multi-pose body dataset and Gaussian attribute residuals, the basic human Gaussian and convolutional neural network models are optimized;

[0193] Generate body Gaussian representation based on the optimized basic human Gaussian and convolutional neural network models;

[0194] The multi-posture body data set includes: a plurality of body images corresponding to different postures, and body parameters corresponding to each body image.

[0195] In some embodiments, the body Gaussian generation module 930 is used to:

[0196] Add the Gaussian attribute residual to the basic human body Gaussian to obtain the added human body Gaussian;

[0197] Performing posture deformation on the added human body Gaussian using body parameters to obtain a posture-deformed Gaussian;

[0198] Based on the body image corresponding to the body parameters, the basic human Gaussian and convolutional neural network models are optimized to adjust the Gaussian after the posture deformation, so that the rendered image of the Gaussian after the posture deformation approaches the body image corresponding to the body parameters.

[0199] In some embodiments, the first fusion module 940 is configured to:

[0200] The lower part of the midline of the Gaussian representation of the head is removed to obtain the Gaussian content of the head; and the upper part of the midline of the Gaussian representation of the body is removed to obtain the Gaussian content of the body;

[0201] The head Gaussian content and the body Gaussian content are combined to obtain a full-body Gaussian representation of the digital human.

[0202] Fig.10 is a structural diagram of a digital human reconstruction device 1000 according to an embodiment of the present disclosure. In some implementations, it further includes:

[0203] An acquisition module 1050 is used to acquire expression parameters and whole body posture parameters in the driving signal;

[0204] A driving module 1060 is used to drive the head Gaussian in the whole body Gaussian representation based on the expression parameters in the driving signal; and to drive the body Gaussian in the whole body Gaussian representation based on the whole body posture parameters in the driving signal;

[0205] The second fusion module 1070 is used to fuse the driven head Gaussian and body Gaussian to obtain a driving result corresponding to the driving signal.

[0206] In some implementations, the digital human reconstruction device 1000 further includes:

[0207] A head data set building module 1080 is used to determine a plurality of head images based on a human body video or image, each head image corresponding to a different expression;

[0208] Based on the facial detail capture algorithm, each head image is calculated to obtain the expression parameters corresponding to the head image;

[0209] A multi-expression head data set is constructed based on the multiple head images and expression parameters corresponding to each head image.

[0210] In some implementations, the digital human reconstruction device 1000 further includes:

[0211] A body data set building module 1090 is used to determine a plurality of body images based on a human body video or image, each body image corresponding to a different posture;

[0212] Based on the human motion capture algorithm and the hand motion capture algorithm, each body image is calculated respectively to obtain the body parameters corresponding to the body image;

[0213] A multi-pose body dataset is constructed based on multiple body images and body parameters corresponding to each body image.

[0214] For the description of specific functions and examples of each module and submodule of the device in the embodiment of the present disclosure, reference can be made to the relevant description of the corresponding steps in the above method embodiment, which will not be repeated here.

[0215] In the technical solution disclosed in the present invention, the acquisition, storage and application of personal information of users involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0216] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0217] Fig.11A schematic block diagram of an example electronic device 1100 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0218] like Fig.11 As shown, the device 1100 includes a computing unit 1101, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1102 or a computer program loaded from a storage unit 1108 into a random access memory (RAM) 1103. In the RAM 1103, various programs and data required for the operation of the device 1100 can also be stored. The computing unit 1101, the ROM 1102, and the RAM 1103 are connected to each other via a bus 1104. An input / output (I / O) interface 1105 is also connected to the bus 1104.

[0219] A number of components in the device 1100 are connected to the I / O interface 1105, including: an input unit 1106, such as a keyboard, a mouse, etc.; an output unit 1107, such as various types of displays, speakers, etc.; a storage unit 1108, such as a disk, an optical disk, etc.; and a communication unit 1109, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1109 allows the device 1100 to exchange data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0220] The computing unit 1101 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1101 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1101 performs the various methods and processes described above, such as detection methods. For example, in some embodiments, the detection method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 1108. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 1100 via ROM 1102 and / or communication unit 1109. When the computer program is loaded into RAM 1103 and executed by the computing unit 1101, one or more steps of the detection method described above may be performed. Alternatively, in other embodiments, the computing unit 1101 may be configured to perform the detection method in any other appropriate manner (e.g., by means of firmware).

[0221] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0222] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0223] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0224] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0225] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0226] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0227] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0228] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the principles of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A digital human reconstruction method, comprising: Building a personalized human body grid based on a human body video or image, wherein the personalized human body grid carries human body feature information in the human body video or image; Extracting a head mesh from the personalized human body mesh, and generating a corresponding head Gaussian representation based on the head mesh; Based on the personalized human body mesh, generating a corresponding body Gaussian representation; The head Gaussian representation and the body Gaussian representation are fused to obtain the full body Gaussian representation of the digital human.

2. The method according to claim 1, wherein: The method of constructing a personalized human body mesh based on a human body video or image includes: Based on the human body video or image, generating a rough human body Gaussian carrying the human body feature information; According to the rough human body Gaussian, a rough human body mesh carrying the human body feature information is obtained; Based on the rough human body mesh, optimizing the initial human body mesh to obtain an optimized human body mesh; The personalized human body mesh is obtained based on the optimized human body mesh and the rough human body mesh.

3. The method according to claim 2, wherein: The generating a rough human body Gaussian carrying the human body feature information based on the human body video or image includes: Constructing the initial human body mesh; Initializing Gaussian point attributes based on the initial human body mesh to obtain an initial human body Gaussian; The initial human body Gaussian is optimized based on the human body video or image to obtain a rough human body Gaussian carrying the human body feature information.

4. The method according to claim 2 or 3, wherein: The step of optimizing the initial human body mesh based on the rough human body mesh comprises: The initial human body mesh is optimized using a three-dimensional semantic mask corresponding to the human body's hands and / or face, and optimization of the human body's hands and / or face is excluded during the optimization process.

5. The method according to claim 2 or 3, wherein: The step of obtaining the personalized human body mesh based on the optimized human body mesh and the rough human body mesh comprises: The human foot region of the rough human body mesh is replaced with the foot region of the optimized human body mesh to obtain the personalized human body mesh; wherein the personalized human body mesh includes the human foot region of the rough human body mesh and the region other than the human foot in the optimized human body mesh.

6. The method according to claim 1, wherein: The generating a corresponding head Gaussian representation based on the head grid includes: Using the head mesh, generating an initial head Gaussian; The initial head Gaussian is optimized using a pre-established multi-expression head data set to obtain the head Gaussian representation; wherein the multi-expression head data set includes: a plurality of head images corresponding to different expressions, and expression parameters corresponding to each of the head images.

7. The method according to claim 6, wherein: The step of using the head mesh to generate an initial head Gaussian comprises: For each triangle in the head mesh, assign a Gaussian point to the triangle and set initial local attributes of the Gaussian point; Determining initial global properties of the Gaussian points based on initial local properties of the Gaussian points and properties of the triangle; An initial head Gaussian is generated based on the initial global properties of each Gaussian point in the head mesh.

8. The method according to claim 6, wherein: The method of optimizing the initial head Gaussian using a pre-established multi-expression head data set to obtain the head Gaussian representation includes: driving the initial head Gaussian based on the expression parameter to generate a target head Gaussian corresponding to the expression parameter; Based on the head image corresponding to the expression parameter, adjusting the target head Gaussian so that the rendered image of the target head Gaussian approaches the head image corresponding to the expression parameter; The head Gaussian representation is determined based on the adjusted target head Gaussian.

9. The method according to claim 1, wherein: The step of generating a corresponding Gaussian representation of the body based on the personalized human body mesh comprises: Based on the personalized human body mesh, generating a basic human body Gaussian; Using the basic human body Gaussian, construct a human body posture graph; Inputting the posture graph into a convolutional neural network model to be optimized to obtain a Gaussian attribute residual output by the convolutional neural network model; Optimizing the basic human Gaussian and the convolutional neural network model based on a pre-established multi-pose body dataset and the Gaussian attribute residuals; Based on the optimized basic human Gaussian and convolutional neural network models, generating the body Gaussian representation; The multi-posture body data set includes: a plurality of body images corresponding to different postures, and body parameters corresponding to each of the body images.

10. The method according to claim 9, wherein: The optimizing the basic human body Gaussian and the convolutional neural network model based on the pre-established multi-posture body data set and the Gaussian attribute residual includes: Adding the Gaussian attribute residual to the basic human body Gaussian to obtain an added human body Gaussian; Performing posture deformation on the added human body Gaussian using the body parameters to obtain a posture-deformed Gaussian; Based on the body image corresponding to the body parameters, the basic human Gaussian and the convolutional neural network model are optimized to adjust the Gaussian after posture deformation so that the rendered image of the Gaussian after posture deformation approaches the body image corresponding to the body parameters.

11. The method according to claim 1, wherein: The step of fusing the head Gaussian representation and the body Gaussian representation to obtain the whole body Gaussian representation of the digital human includes: The lower part of the midline of the Gaussian representation of the head is removed to obtain the Gaussian content of the head; and the upper part of the midline of the Gaussian representation of the body is removed to obtain the Gaussian content of the body; The head Gaussian content and the body Gaussian content are combined to obtain a full-body Gaussian representation of the digital human.

12. The method according to claim 1, further comprising, Obtaining expression parameters and whole body posture parameters in the driving signal; Based on the expression parameters in the driving signal, driving the head Gaussian in the whole body Gaussian representation; and based on the whole body posture parameters in the driving signal, driving the body Gaussian in the whole body Gaussian representation; The driven head Gaussian and body Gaussian are fused to obtain a driving result corresponding to the driving signal.

13. The method according to any one of claims 6 to 8, further comprising: Based on the human body video or image, determining a plurality of head images, each of the head images corresponding to a different expression; Based on a facial detail capture algorithm, each of the head images is calculated to obtain expression parameters corresponding to the head image; The multi-expression head data set is constructed based on the multiple head images and the expression parameters corresponding to each of the head images.

14. The method according to claim 9 or 10, further comprising: Based on the human body video or image, determining a plurality of body images, each of the body images corresponding to a different posture; Based on a human motion capture algorithm and a hand motion capture algorithm, calculating each of the body images respectively to obtain body parameters corresponding to the body images; The multi-posture body dataset is constructed based on the multiple body images and the body parameters corresponding to each of the body images.

15. A digital human reconstruction device, comprising: A mesh construction module, used to construct a personalized human body mesh based on a human body video or image, wherein the personalized human body mesh carries human body feature information in the human body video or image; A head Gaussian generation module, used to extract a head mesh from the personalized human body mesh, and generate a corresponding head Gaussian representation based on the head mesh; A body Gaussian generation module, used to generate a corresponding body Gaussian representation based on the personalized human body mesh; The first fusion module is used to fuse the head Gaussian representation and the body Gaussian representation to obtain the whole body Gaussian representation of the digital human.

16. The device according to claim 15, wherein: The grid building module is used to: Based on the human body video or image, generating a rough human body Gaussian carrying the human body feature information; According to the rough human body Gaussian, a rough human body mesh carrying the human body feature information is obtained; Based on the rough human body mesh, optimizing the initial human body mesh to obtain an optimized human body mesh; The personalized human body mesh is obtained based on the optimized human body mesh and the rough human body mesh.

17. The device according to claim 15, wherein: The grid building module is used to: Constructing the initial human body mesh; Initializing Gaussian point attributes based on the initial human body mesh to obtain an initial human body Gaussian; The initial human body Gaussian is optimized based on the human body video or image to obtain a rough human body Gaussian carrying the human body feature information.

18. The device according to claim 16 or 17, wherein: The grid building module is used to: The initial human body mesh is optimized using a three-dimensional semantic mask corresponding to the human body's hands and / or face, and optimization of the human body's hands and / or face is excluded during the optimization process.

19. The device according to claim 16 or 17, wherein: The grid building module is used to: The human foot region of the rough human body mesh is replaced with the foot region of the optimized human body mesh to obtain the personalized human body mesh; wherein the personalized human body mesh includes the human foot region of the rough human body mesh and the region other than the human foot in the optimized human body mesh.

20. The device according to claim 15, wherein: The head Gaussian generation module is used for: Using the head mesh, generating an initial head Gaussian; The initial head Gaussian is optimized using a pre-established multi-expression head data set to obtain the head Gaussian representation; wherein the multi-expression head data set includes: a plurality of head images corresponding to different expressions, and expression parameters corresponding to each of the head images.

21. The device according to claim 20, wherein: The head Gaussian generation module is used for: For each triangle in the head mesh, assign a Gaussian point to the triangle and set initial local attributes of the Gaussian point; Determining initial global properties of the Gaussian points based on initial local properties of the Gaussian points and properties of the triangle; An initial head Gaussian is generated based on the initial global properties of each Gaussian point in the head mesh.

22. The device according to claim 20, wherein: The head Gaussian generation module is used for: driving the initial head Gaussian based on the expression parameter to generate a target head Gaussian corresponding to the expression parameter; Based on the head image corresponding to the expression parameter, adjusting the target head Gaussian so that the rendered image of the target head Gaussian approaches the head image corresponding to the expression parameter; The head Gaussian representation is determined based on the adjusted target head Gaussian.

23. The device according to claim 15, wherein: The body Gaussian generation module is used to: Based on the personalized human body mesh, generating a basic human body Gaussian; Using the basic human body Gaussian, construct a human body posture graph; Inputting the posture graph into a convolutional neural network model to be optimized to obtain a Gaussian attribute residual output by the convolutional neural network model; Optimizing the basic human Gaussian and the convolutional neural network model based on a pre-established multi-pose body dataset and the Gaussian attribute residuals; Based on the optimized basic human Gaussian and convolutional neural network models, generating the body Gaussian representation; The multi-posture body data set includes: a plurality of body images corresponding to different postures, and body parameters corresponding to each of the body images.

24. The device according to claim 23, wherein: The body Gaussian generation module is used to: Adding the Gaussian attribute residual to the basic human body Gaussian to obtain an added human body Gaussian; Performing posture deformation on the added human body Gaussian using the body parameters to obtain a posture-deformed Gaussian; Based on the body image corresponding to the body parameters, the basic human Gaussian and the convolutional neural network model are optimized to adjust the Gaussian after posture deformation so that the rendered image of the Gaussian after posture deformation approaches the body image corresponding to the body parameters.

25. The device according to claim 15, wherein: The first fusion module is used to: The lower part of the midline of the Gaussian representation of the head is removed to obtain the Gaussian content of the head; and the upper part of the midline of the Gaussian representation of the body is removed to obtain the Gaussian content of the body; The head Gaussian content and the body Gaussian content are combined to obtain a full-body Gaussian representation of the digital human.

26. The apparatus of claim 15, further comprising: An acquisition module, used to acquire expression parameters and whole body posture parameters in the driving signal; A driving module, configured to drive the head Gaussian in the whole body Gaussian representation based on the expression parameter in the driving signal; and driving a body Gaussian in the whole body Gaussian representation based on the whole body posture parameter in the driving signal; The second fusion module is used to fuse the driven head Gaussian and body Gaussian to obtain a driving result corresponding to the driving signal.

27. The apparatus according to any one of claims 20 to 22, further comprising: A head data set building module, used to determine a plurality of head images based on the human body video or image, each of the head images corresponding to a different expression; Based on a facial detail capture algorithm, each of the head images is calculated to obtain expression parameters corresponding to the head image; The multi-expression head data set is constructed based on the multiple head images and the expression parameters corresponding to each of the head images.

28. The apparatus according to claim 23 or 24, further comprising: A body data set establishment module, used to determine a plurality of body images based on the body video or image, each of the body images corresponding to a different posture; Based on the human motion capture algorithm and the hand motion capture algorithm, each of the body images is calculated respectively to obtain the body parameters corresponding to the body image; based on the multiple body images and the body parameters corresponding to each of the body images, the multi-posture body data set is constructed.

29. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 14.

30. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-14.

31. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 14.

Images