Digital human generation method and device and storage medium

Abstract

The embodiment of the invention provides a digital human generation method and device and a storage medium. According to the embodiment of the invention, after the expression driving signal and the limb driving signal of the target object are obtained, the expression driving signal and the limb driving signal can be fused, then, on one hand, the Gaussian parameter is extracted from the fused signal to obtain the Gaussian parameter variation, and on the other hand, the Gaussian parameter variation is obtained. According to the method, Gaussian parameters of a preset standard object in a standard action posture are obtained from a skeletal animation model, and then Gaussian parameters of a target object in a target action posture can be calculated according to the variable quantity of the Gaussian parameters and the Gaussian parameters of the standard object in the standard action posture. Rendering a two-dimensional image of the target object under the target action posture, and obtaining a digital human corresponding to the target object; according to the scheme, the generation efficiency of the digital human can be improved, and the rendering quality can be improved.

Description

Technical Field

[0001] The present invention relates to the field of computer control technology, and in particular to a method, device and storage medium for generating a digital human. Background Art

[0002] Digital human refers to a digital human image (or digital animal image) that is close to human image (or other animal image) and created by digital technology. It can be either a physical digital human or a virtual digital human. Among them, physical digital human is also called real digital human (Life-like Animatable Avatars), which refers to the digital entity generated by digitizing the appearance, voice and action of real people (or other real animals) through digital technology (that is, the digital representation of real people or animals); while virtual digital human is a digital entity completely based on computer simulation and generation. With the development of artificial intelligence (AI) technology, digital human, especially real digital human, is increasingly widely used in technologies such as virtual companionship (Virtual-company) and extended reality (Extended Reality).

[0003] In the prior art, the generation of digital humans generally adopts the mesh expression method, that is, the mesh shape of the digital human to be generated is first determined based on each posture of the human body, and then based on these mesh shapes, the mesh surface is expressed using materials, such as shadows, textures, etc., so as to render static three-dimensional images corresponding to each posture, and then, these static three-dimensional images are combined to obtain the desired digital human.

[0004] In the process of studying and using the prior art, the inventor of the present invention found that since the prior art needs to determine the mesh shape and render each posture and expression of the digital human separately, if a full-body talking avatar is to be generated, a series of complex and repetitive calculations and renderings are required, which is not only inefficient but also has poor overall rendering quality. Summary of the invention

[0005] The embodiments of the present invention provide a method, device and storage medium for generating a digital human, which can improve the generation efficiency and rendering quality of a digital human.

[0006] The embodiment of the present invention provides a method for generating a digital human, comprising:

[0007] Establish a three-dimensional model of the target object in the target action posture through the skeletal animation model;

[0008] Acquire an expression driving signal of the target object, and acquire a limb driving signal of the target object in a target action posture from the three-dimensional model;

[0009] fusing the expression drive signal and the limb drive signal to obtain a fused signal;

[0010] Extracting Gaussian parameters from the fused signal to obtain Gaussian parameter variation;

[0011] Obtaining Gaussian parameters of a preset standard object in a standard action posture from the skeletal animation model;

[0012] Calculating the Gaussian parameters of the target object in the target action posture according to the Gaussian parameter variation and the Gaussian parameters of the standard object in the standard action posture;

[0013] According to the Gaussian parameters of the target object in the target action posture, a two-dimensional image of the target object in the target action posture is rendered to obtain a digital human corresponding to the target object.

[0014] Accordingly, the embodiment of the present application further provides a digital human generation device, including:

[0015] A building unit, used for building a three-dimensional model of a target object in a target action posture through a skeletal animation model;

[0016] An acquisition unit, used to acquire an expression drive signal of a target object, and acquire a limb drive signal of the target object in a target action posture from the three-dimensional model;

[0017] A fusion unit, used for fusing the expression drive signal and the limb drive signal to obtain a fused signal;

[0018] An extraction unit, used for extracting Gaussian parameters from the fused signal to obtain Gaussian parameter variation;

[0019] A calculation unit, used for obtaining Gaussian parameters of a preset standard object in a standard action posture from the skeletal animation model, and calculating Gaussian parameters of the target object in a target action posture according to the Gaussian parameter variation and the Gaussian parameters of the standard object in the standard action posture;

[0020] The rendering unit is used to render a two-dimensional image of the target object in the target action posture according to the Gaussian parameters of the target object in the target action posture, so as to obtain a digital human corresponding to the target object.

[0021] In addition, an embodiment of the present application further provides a storage medium, wherein the storage medium stores a plurality of instructions, and the instructions are suitable for loading by a processor to execute the steps in any one of the digital human generation methods provided in the embodiments of the present application.

[0022] After acquiring the expression drive signal and the limb drive signal of the target object, the embodiment of the present application can fuse the expression drive signal and the limb drive signal, and then, on the one hand, extract the Gaussian parameters from the fused signals to obtain the Gaussian parameter variation, and on the other hand, obtain the Gaussian parameters of the preset standard object in the standard action posture from the skeletal animation model, and then, according to the Gaussian parameter variation and the Gaussian parameters of the standard object in the standard action posture, calculate the Gaussian parameters of the target object in the target action posture, and render the two-dimensional image of the target object in the target action posture accordingly to obtain the digital human corresponding to the target object; since the scheme only needs to input the expression drive signal and the limb drive signal of the target object to render the digital human corresponding to the target object, without having to determine the grid shape and render it separately for each posture and each expression of the target object, therefore, the amount of calculation and the number of rendering times can be greatly reduced, and the generation efficiency can be improved. Moreover, since the rendering is based on the Gaussian parameters, compared with the technology based on the grid rendering, it can add more rendering details and greatly improve the rendering quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0024] Figure 1 is a flow chart of a digital human generation method provided in an embodiment of the present application;

[0025] Figure 2 It is an example diagram of the UV expansion diagram provided in the embodiment of the present application;

[0026] Figure 3 is a schematic diagram of the processing flow of the limb drive signal in the embodiment of the present application;

[0027] Figure 4 is a schematic diagram of the processing flow of the expression drive signal in the embodiment of the present application;

[0028] Figure 5 is an example diagram of a standard object in a standard posture in an embodiment of the present application;

[0029] Figure 6 is another flow chart of the digital human generation method provided in the embodiment of the present application;

[0030] Figure 7 It is a schematic diagram of the processing flow of the digital human generation network provided in the embodiment of the present application;

[0031] Figure 8 It is a structural schematic diagram of a digital human generation device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0032] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0033] The embodiments of the present application provide a method, device and storage medium for generating a digital human, which are described in detail below.

[0034] This embodiment will be described from the perspective of a digital human generation device. In specific implementation, the digital human generation device can be integrated into a device such as a terminal or a server, wherein the terminal can be a personal computer (Personal Computer), a laptop computer, a tablet computer or other smart terminals.

[0035] A digital human generation method comprises: obtaining an expression drive signal of a target object and a limb drive signal of the target object in a target action posture (the limb drive signal can be obtained from a three-dimensional model of the target object in the target action posture, and the three-dimensional model can be established by a skeleton animation model), fusing the expression drive signal and the limb drive signal to obtain a fused signal; then, on the one hand, extracting Gaussian parameters from the fused signal to obtain Gaussian parameter variation, and on the other hand, obtaining Gaussian parameters of a preset standard object in a standard action posture from the skeleton animation model; then, the Gaussian parameters of the target object in the target action posture can be calculated according to the Gaussian parameter variation and the Gaussian parameters of the standard object in the standard action posture, and according to the Gaussian parameters of the target object in the target action posture, a two-dimensional image of the target object in the target action posture is rendered to obtain a digital human corresponding to the target object.

[0036] For example, Figure 1 As shown, the specific process of the digital human generation method can be as follows:

[0037] 101. Obtain an expression driving signal of a target object and a body driving signal of the target object in a target action posture.

[0038] Among them, the expression driving signal refers to the driving signal of the facial expression, and there can be various types. For example, the traditional 3DMM (3D Morphable Model) can be used as the expression driving signal, or a two-dimensional facial image can be used as the expression driving signal, and so on.

[0039] Similarly, the limb driving signal refers to the driving signal of the limb action, and there may be multiple types of limb driving signals. For example, the body posture parameters of the three-dimensional model may be used as the limb driving signal, etc. That is to say, a three-dimensional model of the target object in the target action posture may be first established through a skeletal animation model, and then the limb driving signal of the target object in the target action posture may be obtained from the three-dimensional model; that is, the step of "obtaining the expression driving signal of the target object and the limb driving signal of the target object in the target action posture" may specifically include:

[0040] A three-dimensional model of the target object in a target action posture is established through a skeletal animation model; an expression driving signal of the target object is obtained, and a limb driving signal of the target object in the target action posture is obtained from the three-dimensional model.

[0041] For example, the body posture parameters of each body part of the target object in the target action posture can be obtained from the three-dimensional model, wherein these body posture parameters can be used as limb driving signals of the target object in the target action posture.

[0042] It should be noted that, in this application, for the convenience of description, the skeletal animation model of the target object is referred to as a “three-dimensional model”, and the tool for establishing the skeletal animation model of the target object is referred to as a “skeletal animation model”.

[0043] It should also be noted that in the embodiments of the present application, the target object may include a human or other animals, such as a monkey, a dog or a cat, etc. The target action posture may be any posture, which may be determined according to the needs of the actual application or the preferences of the target object, and the "body part" may be determined according to the body structure of the target object. For example, taking the target object as a human, the "body part" may be the knee, foot, shoulder, spine, etc., which will not be elaborated here.

[0044] 102. Fuse the expression drive signal and the body drive signal to obtain a fused signal.

[0045] For example, taking the limb driving signal as the body posture parameter, at this time, the encoder in the preset digital human generation network can be used to fuse the expression driving signal and the body posture parameter to obtain a fused signal.

[0046] The digital human generation network may include an encoder and a Gaussian parameter decoder. Optionally, the encoder may be set according to the needs of the actual application. For example, taking the encoder in the digital human generation network including a body feature encoder, a local feature encoder, a global feature encoder and an expression encoder as an example, the step of "using the encoder in the preset digital human generation network to fuse the expression drive signal and the limb drive signal to obtain a fused signal" may include steps S1 to S7, as follows:

[0047] S1. Obtain, from the three-dimensional model, a UV unfolding diagram of each body part of the target object in a target action posture, a skinning weight of a bone to a vertex, and a vertex position diagram of the three-dimensional model.

[0048] UV is the abbreviation of texture coordinates, and the UV expansion map refers to the skin of the three-dimensional model. For example, see Figure 2 , the picture shows the UV unfolding diagram of the human face, left knee, left foot, left shoulder, spine, jaw, left wrist and other body parts.

[0049] Generally, a 3D model contains a skeleton and a surface mesh (mesh for short), and the mesh includes vertices, lines, and mesh faces. The influence of the skeleton on the vertex is called skinning. For example, the skinning weight of the left-hand skeleton on the left-hand vertex is very high, but the skinning weight of the right-hand vertex is 0, and so on. In addition, the mesh generally corresponds to a UV unfolding map for use with texture mapping. Therefore, the UV unfolding map of each body part of the target object in the target action posture and the skinning weight of the skeleton on the vertex can be directly obtained from the 3D model, and the vertex position map can be generated by obtaining the information of each vertex, that is, the step of "obtaining the vertex position map of the 3D model" can include:

[0050] By obtaining the information of each vertex of the three-dimensional model when the target object is in the target action posture, and associating the information of each vertex of the three-dimensional model with the corresponding vertex in the UV unfolded map (equivalent to filling the vertex position of the mesh into the UV unfolded map), the pose vertex correspondence (posed-vertex map) of the three-dimensional model can be obtained. For the convenience of description, in the embodiment of the present application, the "pose vertex correspondence of the three-dimensional model" is referred to as the vertex position map of the three-dimensional model.

[0051] The size of the vertex position map can be set according to the needs of actual applications. For example, it can generally be set to 512x512x3, and so on.

[0052] S2. Using a body feature encoder, a posture feature matrix of the target object in a target action posture is generated according to the body posture parameters, the skin weights of the bones to the vertices, and the UV unfolding diagram. For example, the specific steps are as follows:

[0053] Through the body feature encoder, the body posture parameters of each body part, the skinning weights of the bones to the vertices, and the UV unfolding map are encoded to obtain posture feature maps of multiple body parts. By superimposing the posture feature maps of the multiple body parts, the posture feature matrix of the target object in the target action posture can be obtained.

[0054] There are many ways to implement the step of "encoding the body posture parameters of each body part, the skinning weights of the bones to the vertices, and the UV unfolding map through the body feature encoder to obtain the posture feature maps of multiple body parts", for example, the specific implementation method is as follows:

[0055] Through the body feature encoder, the body posture parameters of each body part are filled into the UV expansion map of the corresponding body part to obtain the UV expansion map to be processed corresponding to each body part; according to the skinning weight of the bone to the vertex, the UV expansion map to be processed is converted into a mask to obtain the multi-layer feature value corresponding to each body part; the multi-layer feature value corresponding to each body part is encoded respectively to obtain the posture feature maps of multiple body parts.

[0056] For example, Figure 3 As shown, specifically, the body posture parameters of each body part can be filled into the UV unfolding map of the corresponding body part through the body feature encoder to obtain the UV unfolding map to be processed corresponding to each body part, and then the skinning weights of each bone to the vertices are filled into these UV unfolding maps to be processed (the UV unfolding map corresponds to the vertices of the mesh), and converted into a mask (mash), that is, the mask is 1 where there is a weight, and the mask is 0 where there is no weight, and then the multi-layer feature value corresponding to each body part is obtained. After that, the convolutional neural network in the body feature encoder can be used for encoding to obtain the posture feature map corresponding to each body part (that is, posture feature maps of multiple body parts will be obtained at this time).

[0057] Generally, in computers, these posture feature graphs can be represented using a two-dimensional matrix. For example, the posture feature graph a can be represented as The posture feature map b can be expressed as Etc. Therefore, by simply superimposing these posture feature maps, a three-dimensional feature matrix can be obtained. For example, the three-dimensional matrix after superimposing the above posture feature maps a and b is That is to say, the body feature encoder will eventually output a three-dimensional feature matrix. For the convenience of description, in the embodiment of the present application, the three-dimensional feature matrix output by the body feature encoder is called "the posture feature matrix of the target object in the target action posture", or posture feature matrix for short.

[0058] S3. Feature encode the vertex position map through a local feature encoder to obtain a local vertex feature matrix.

[0059] For example, Figure 3 As shown, the vertex position map can be input into a local feature encoder, which directly encodes the vertex position map into a feature matrix. For example, it can be a feature matrix of size 64x64x32, which is the local vertex feature matrix.

[0060] Among them, the local feature encoder can be implemented by using a convolutional neural network. The number of layers and network parameters of the convolutional neural network can be determined according to the needs of actual applications and are not limited here.

[0061] S4. Perform feature encoding on the vertex position graph through a global feature encoder to obtain a global vertex feature matrix.

[0062] For example, Figure 3 As shown, the vertex position map can be input into the global feature encoder, and the global feature encoder encodes the vertex position map into a feature vector, for example, a feature vector of size 1x1x2048, and then decodes the feature vector into a feature matrix, for example, decoding a feature vector of size 1x1x2048 into a feature matrix of size 64x64x32, which is the global vertex feature matrix.

[0063] Similar to the local feature decoder, the global feature encoder can also be implemented using a convolutional neural network. The number of layers and network parameters of the convolutional neural network can be determined according to the needs of the actual application. However, when setting up, the number of network layers of the global feature decoder should be more than that of the local feature decoder, so that in addition to the differences in the learned features of the two feature decoders, the global feature encoder can pay more attention to global information.

[0064] It should be noted that steps S2, S3 and S4 may be performed in any order.

[0065] S5. Merge the posture feature matrix, the local vertex feature matrix, and the global vertex feature matrix to obtain a merged feature matrix.

[0066] For example, Figure 3As shown, the posture feature matrix, the local vertex feature matrix, and the global vertex feature matrix can be stacked up in one direction to form a large three-dimensional matrix, which is the merged feature matrix.

[0067] S6. Perform feature encoding on the expression drive signal through an expression encoder to obtain an encoded facial feature vector.

[0068] For instance, taking the use of two-dimensional facial images as expression driving signals as an example, multiple two-dimensional facial images can be encoded through an expression encoder to obtain vectors corresponding to the multiple two-dimensional facial images (the data contained in these vectors represents the expression information of the characters in the existing images). Then, the vectors corresponding to the multiple two-dimensional facial images are randomly weighted averaged to obtain the encoded facial feature vector.

[0069] For example, see Figure 4 , taking two-dimensional facial images A, B, C and D as an example, after the two-dimensional facial images A, B, C and D are input into the expression encoder, the expression encoder will encode them respectively and output vector 1, vector 2, vector 3 and vector 4 respectively, wherein vector 1 contains the expression information of the person in the two-dimensional facial image A, vector 2 contains the expression information of the person in the two-dimensional facial image B, vector 3 contains the expression information of the person in the two-dimensional facial image C, and so on, vector 4 also contains the expression information of the person in the two-dimensional facial image D. Vector 1, vector 2, vector 3 and vector 4 are randomly weighted averaged, and the result is the encoded facial feature vector.

[0070] S7. Fusing the merged feature matrix and the encoded facial feature vector to obtain a fused signal.

[0071] Among them, the operation of fusing the merged feature matrix and the encoded facial feature vector into one signal is also called the injection operation of the facial signal. Specifically, the injection operation is to replace the data in the encoded facial feature vector with the original data used to represent facial movements in the three-dimensional human model (i.e., skeletal animation). It should be noted that the three-dimensional human model itself has some data used to represent facial movements, such as the jaw posture parameters and eye posture parameters in SMPL-X (Skinned Multi-PersonLinear-X, a skeleton-driven parameterized human model). These data used to represent facial movements can be directly read from the model parameters. After the operation of steps S1 to S5, these data used to represent facial movements are just located in the upper left part of the merged feature matrix (the data in the other parts are the parameters representing the body posture in the three-dimensional human model). Therefore, at this time, it is only necessary to replace the data in the upper left part of the merged feature matrix with the data in the encoded facial feature vector.

[0072] Of course, since the merged feature matrix is ​​a three-dimensional matrix, the encoded facial feature vector also needs to be converted into a three-dimensional matrix, so that this part of the data can directly replace the data in the upper left part of the merged feature matrix. In other words, the step of "fusing the merged feature matrix and the encoded facial feature vector to obtain a fused signal" can specifically include:

[0073] The encoded facial feature vector is converted into a three-dimensional matrix to obtain a facial feature matrix, and the data used to represent facial movements in the merged feature matrix (such as the data in the upper left part) is replaced with the data in the facial feature matrix to obtain a fused signal.

[0074] Optionally, there are multiple ways to convert the encoded facial feature vector into a three-dimensional matrix. For example, the encoded facial feature vector can be processed by a fully connected layer (FC) and a small convolutional decoder to convert the encoded facial feature vector into a three-dimensional matrix, and so on.

[0075] 103. Extract Gaussian parameters from the fused signal to obtain Gaussian parameter variation.

[0076] For example, the Gaussian parameter decoder in the digital human generation network can be preset to extract the Gaussian parameter from the fused signal to obtain the Gaussian parameter variation; specifically, as follows:

[0077] The fused signal is passed to the Gaussian parameter decoder, which is a convolution decoder. Through the Gaussian parameter decoder, the fused signal can be converted from a three-dimensional matrix into a non-vector representation (that is, information with different meanings in the three-dimensional matrix is ​​separated), thereby converting the original three-dimensional matrix output result into a Gaussian parameter change, such as position change parameter Δμ, rotation change parameter Δq, and scaling change parameter Δs, etc. Of course, other Gaussian parameters can also be extracted at this time, such as Gaussian opacity parameter o and Gaussian color parameter c, etc.

[0078] 104. Obtain Gaussian parameters of a preset standard object in a standard action posture from the skeletal animation model.

[0079] The standard object refers to the standard character in the standard action posture of the skeletal animation model. Figure 5 As shown, taking the standard action posture as a T-shaped posture as an example, at this time, the Gaussian parameters of the standard object in the T-shaped posture can be obtained from the skeletal animation model, including:

[0080] Base position parameter Refers to the position of the mesh of the character's 3D model when it is rendered to UV when the character is in a T-shaped pose (i.e., each valid UV pixel represents a 3D Gaussian initial point in orthogonal space);

[0081] Base rotation parameters This parameter can be obtained by matching the first axis of each 3D Gaussian initial point to the mesh surface and the third axis to the mesh normal;

[0082] Basic scaling parameters This parameter is used to indicate the basic size of the Gaussian.

[0083] 105. Calculate the Gaussian parameters of the target object in the target action posture according to the Gaussian parameter variation and the Gaussian parameters of the standard object in the standard action posture; for example, the specific method may be as follows:

[0084] The Gaussian parameters of the standard object in the standard action posture are adjusted according to the Gaussian parameter change to obtain the Gaussian parameters of the target object in the standard action posture; according to the body posture parameters and the Gaussian parameters of the target object in the standard action posture, a linear blending skinning (LBS) algorithm is used to calculate the Gaussian parameters of the target object in the target action posture.

[0085] For example, taking the Gaussian parameter variation including position variation parameter Δμ, rotation variation parameter Δq, and scaling variation parameter Δs as an example, at this time, the Gaussian parameters of the standard object in the standard action posture, such as the basic position parameter Δμ, the rotation variation parameter Δq, and the scaling variation parameter Δs, can be respectively calculated based on the position variation parameter Δμ, the rotation variation parameter Δq, and the scaling variation parameter Δs. Basic rotation parameters And basic scaling parameters Adjustments are made to obtain Gaussian parameters (indicating the position, rotation and size of the Gaussian) of the target object in the standard action posture, thereby achieving the purpose of converting the standard object in the standard action posture into the target object in the standard action posture, that is, at this time, the three-dimensional model of the standard object in the standard action posture has been converted into the three-dimensional model of the target object in the standard action posture. Next, the body posture parameters can be passed into the three-dimensional model of the target object in the standard action posture, so that it can be converted from the standard action posture to the posture represented by the body posture parameters (i.e., the target action posture). The process can be expressed by the calculation formula as follows:

[0086]

[0087] Among them, μ pis the position parameter of the target object in the target action posture, q p is the rotation parameter of the target object in the target action posture, and s is the scaling parameter of the target object in the target action posture; LBS Rt Represents the linear blend skinning algorithm affected by vertex rotation and displacement, LBS R It represents the linear blend skinning algorithm affected by vertex rotation, θ is the body posture parameter, and w is the center of gravity interpolation skin weight of the skeletal animation model, which can be obtained from the skeletal animation model.

[0088] 106. Render a two-dimensional image of the target object in the target action posture according to the Gaussian parameters of the target object in the target action posture (such as position parameters, rotation parameters, and scaling parameters of the target object in the target action posture) to obtain a digital human corresponding to the target object.

[0089] That is to say, after the operation of step 105, a three-dimensional digital human will eventually be obtained. In order to display the digital human on the screen, the three-dimensional digital human needs to be projected onto a two-dimensional plane. Therefore, it needs to be rendered into a two-dimensional image at this time. In this way, the digital human corresponding to the target object can be seen on the screen.

[0090] There are many ways to render, for example, it can be achieved through the rendering process of standard 3DGS (3D Gaussian Splatting, a 3D graphics rendering technology). The so-called 3DGS mainly uses Gaussian spheres to represent three-dimensional models. Although these Gaussian spheres are discrete in space, they are continuous and differentiable inside. Therefore, this technology can achieve a balance between rendering quality and speed, not only overcoming the limitations of traditional 3D model representation methods in terms of noise and rendering quality, but also greatly improving rendering speed.

[0091] Optionally, in the embodiment of the present application, the digital human generation network can directly adopt a trained generation network, or the digital human generation device can train itself. That is, before the step of "using the encoder in the preset digital human generation network to fuse the expression drive signal and the limb drive signal to obtain a fused signal", the digital human generation method can also include:

[0092] The preset generation network is trained to obtain a digital human generation network. For example, the specific training method can be as follows:

[0093] (1) Collect multiple two-dimensional images corresponding to training samples to obtain real images.

[0094] For example, two-dimensional images corresponding to a plurality of training samples may be collected by using an image acquisition device such as a camera or a video camera.

[0095] For example, taking Zhang San as the training sample, at this time, while Zhang San makes various body movements and expressions, Zhang San's body can be photographed using a camera or video camera and other equipment, and multiple two-dimensional body images corresponding to Zhang San can be obtained, and these two-dimensional body images can be used as Zhang San's real images.

[0096] It should be noted that different types of training samples can be collected according to the different requirements of the required digital human generation network. For example, if the digital human generation network is mainly for humans, then the training samples are humans, and if the digital human generation network is mainly for other animals such as dogs, then the training samples are dogs, and so on, which will not be elaborated here.

[0097] (2) A three-dimensional model of the training sample under the target action posture is established through the skeletal animation model to obtain the three-dimensional model of the training sample.

[0098] (3) Obtaining the expression driving signal of the training sample, and obtaining the limb driving signal of the training sample under the target action posture from the three-dimensional model of the training sample.

[0099] Among them, the expression driving signal can use not only the traditional 3DMM, but also the two-dimensional facial image. And the limb driving signal can use the parameters such as the body posture parameters of the three-dimensional model, that is, the step of "obtaining the limb driving signal of the training sample under the target action posture from the three-dimensional model of the training sample" can be specifically:

[0100] From the three-dimensional model, the body posture parameters of each body part of the training sample under the target action posture are obtained, wherein these body posture parameters can be used as limb driving signals of the training sample under the target action posture.

[0101] (4) The encoder in the preset convolutional neural network is used to fuse the expression drive signal and the limb drive signal of the training sample to obtain the fused signal of the training sample.

[0102] For example, taking the limb driving signal as the body posture parameter, at this time, the encoder in the preset convolutional neural network can be used to fuse the expression driving signal and the body posture parameter of the training sample to obtain the fused signal of the training sample.

[0103] Optionally, the encoder can be set according to the needs of actual applications. For example, taking the encoder in the convolutional neural network including a body feature encoder, a local feature encoder, a global feature encoder and an expression encoder as an example, the step of "using the encoder in the preset convolutional neural network to fuse the expression drive signal and the body posture parameter of the training sample to obtain the fused signal of the training sample" may include:

[0104] From the three-dimensional model of the training sample, the UV expansion map of each body part of the training sample under the target action posture, the skin weight of the bones to the vertices, and the vertex position map of the three-dimensional model are obtained; using the body feature encoder, the posture feature matrix of the training sample under the target action posture is generated according to the body posture parameters, the skin weight of the bones to the vertices and the UV expansion map; the vertex position map is feature-encoded by the local feature encoder to obtain the local vertex feature matrix; the vertex position map is feature-encoded by the global feature encoder to obtain the global vertex feature matrix; the posture feature matrix, the local vertex feature matrix, and the global vertex feature matrix are merged to obtain the merged feature matrix of the training sample; the expression drive signal of the training sample is feature-encoded by the expression encoder to obtain the encoded facial feature vector of the training sample; the merged feature matrix and the encoded facial feature vector of the training sample are fused to obtain the fused signal of the training sample.

[0105] (5) The Gaussian parameters are extracted from the fused signals of the training samples through the Gaussian parameter decoder in the convolutional neural network to obtain the Gaussian parameter changes of the training samples, such as the position change parameter Δμ, the rotation change parameter Δq, and the scaling change parameter Δs. Of course, other Gaussian parameters can also be extracted at this time, such as the Gaussian opacity parameter o and the Gaussian color parameter c, etc.

[0106] (6) Obtain Gaussian parameters of a preset standard object in a standard action posture from the skeletal animation model.

[0107] For example, taking the standard action posture as a T-shaped posture as an example, at this time, the Gaussian parameters of the standard object under the T-shaped posture can be obtained from the skeletal animation model, such as the basic position parameters Basic rotation parameters And basic scaling parameters wait.

[0108] (7) Calculating the Gaussian parameters of the training sample in the target action posture according to the Gaussian parameter variation of the training sample and the Gaussian parameters of the standard object in the standard action posture; for example, the specific method may be as follows:

[0109] According to the change of Gaussian parameters of the training sample, the Gaussian parameters of the standard object in the standard action posture are adjusted to obtain the Gaussian parameters of the training sample in the standard action posture; according to the body posture parameters of the training sample and the Gaussian parameters of the training sample in the standard action posture, the linear mixed skinning algorithm is used to calculate the Gaussian parameters of the training sample in the target action posture.

[0110] (8) Render a two-dimensional image of the training sample under the target action posture according to the Gaussian parameters of the training sample under the target action posture to obtain a predicted image.

[0111] For example, the two-dimensional image of the training sample in the target action posture can be rendered using standard 3DGS according to the Gaussian parameters of the training sample in the target action posture to obtain a predicted image.

[0112] The specific implementation of the above steps is similar to that of the above target object. Please refer to the previous embodiment for details, and will not be repeated here.

[0113] (9) Using a preset loss function, the convolutional neural network is converged according to the real image and the predicted image to obtain a digital human generation network.

[0114] For example, the difference between the predicted image and the real image, the learned perceptual image patch similarity (LPIPS, Learned Perceptual Image Patch Similarity) between the predicted image and the real image, and the structural similarity (SSIM, Structural Similarity IndexMeasurement) between the predicted image and the real image can be calculated. Then, the loss function is used to converge the convolutional neural network based on the difference, learned perceptual image patch similarity, structural similarity and position change parameters (the value of the loss function is calculated, and the calculated value of the loss function is used for iteration. When the value of the loss function is small enough, such as less than a preset value, the iteration is ended, and the training of the digital human model is completed), and the digital human generation network can be obtained.

[0115] Among them, the loss function can be set according to the needs of the actual application, for example, it can be set as follows:

[0116]

[0117] in, is the difference between the predicted image and the real image, and It is used to measure the predicted image

[0118] The parameters of similarity with the real image, To learn perceptual image patch similarity (LPIPS, Learned Perceptual Image Patch Similarity), is the structural similarity; Δμ is the position change parameter (i.e., pose-dependent position).

[0119] is a regularization term used to prevent the size of the three-dimensional Gaussian ellipsoid from becoming too large during training; That is, the size of the three-dimensional Gaussian ellipsoid, when More than 10 times When the average value is Set as The absolute value of the value is used as a penalty term for the three-dimensional Gaussian ellipsoid being too large. Less than or equal to 10 times When the average value is Set to 0.

[0120] λ ssim , lpips , μ and λ s are all training hyperparameters that can be set by yourself. In the embodiment of this application, λ ssim Set to 0.2, λ lpips Set to 0.1, λ μ Set to 0.001, λ s Set to 1.0.

[0121] As can be seen from the above, after acquiring the expression drive signal and the limb drive signal of the target object, the embodiment can fuse the expression drive signal and the limb drive signal, and then, on the one hand, extract the Gaussian parameters from the fused signals to obtain the Gaussian parameter variation, and on the other hand, obtain the Gaussian parameters of the preset standard object in the standard action posture from the skeletal animation model, and then, according to the Gaussian parameter variation and the Gaussian parameters of the standard object in the standard action posture, the Gaussian parameters of the target object in the target action posture can be calculated, and the two-dimensional image of the target object in the target action posture can be rendered accordingly to obtain the digital human corresponding to the target object; since the scheme only needs to input the expression drive signal and the limb drive signal of the target object to render the digital human corresponding to the target object, and there is no need to determine the grid shape and render it separately for each posture and each expression of the target object, therefore, the amount of calculation and the number of rendering times can be greatly reduced, and the generation efficiency can be improved. Moreover, since the rendering is based on the Gaussian parameters, compared with the technology based on the grid rendering, it can add more rendering details and greatly improve the rendering quality.

[0122] According to the method described in the previous embodiment, the following will take the example that the digital human generation device is specifically integrated into an electronic device and a trained digital human generation network is provided in the digital human generation device as an example for further detailed description.

[0123] Among them, the digital human generation network includes an encoder and a decoder, the encoder includes an expression encoder, a body feature encoder, a local feature encoder and a global feature encoder, the decoder is a Gaussian parameter decoder, and the training method of the digital human generation network can be found in the previous embodiment and will not be repeated here.

[0124] In addition, for the convenience of description, in this embodiment, a two-dimensional facial image is used as an expression driving signal, and a body posture parameter is used as a limb driving signal as an example for explanation. It should be understood that other types of signals can also be used as expression driving signals and limb driving signals.

[0125] like Figure 6 As shown, a method for generating a digital human, the specific process can be as follows:

[0126] 201. The electronic device establishes a three-dimensional model of the target object in the target action posture through the skeletal animation model, and obtains body posture parameters (i.e., limb drive signals) of each body part of the target object in the target action posture from the three-dimensional model, and then executes step 202.

[0127] For example, if the target object is Zhang San, and the target action posture is Zhang San's "standing up and raising his left hand" action (see Figure 7 ), then the electronic device can obtain the body posture parameters of each part of Zhang San in the posture of "standing and raising his left hand" from the three-dimensional model, such as hands, knees, feet, shoulders and spine, and these body posture parameters are the limb drive signals.

[0128] 202. The electronic device obtains, from the three-dimensional model, a UV unfolding diagram of each body part of the target object in the target action posture, a skinning weight of the bones to the vertices, and a vertex position diagram of the three-dimensional model.

[0129] For example, Figure 7 As shown, taking the target action posture of "standing and raising the left hand" as an example, at this time, the electronic device can obtain the UV unfolding map of each body part in the posture of "standing and raising the left hand", such as the UV unfolding map of the left knee, the UV unfolding map of the left foot, the UV unfolding map of the left shoulder, the UV unfolding map of the left wrist, and the UV unfolding map of the jaw, etc.; and obtain the skinning weights of the vertices of each bone and the vertex position map of the three-dimensional model.

[0130] Among them, the vertex position map can be obtained by filling the vertex information (the vertex information can be obtained by filling the body posture parameters into the skeletal animation model) into the UV expansion map. For details, please refer to the previous embodiment and will not be repeated here. The size of the vertex position map can be set according to the needs of actual applications. For example, it can generally be set to 512x512x3, etc.

[0131] 203. The electronic device uses a body feature encoder to generate a posture feature matrix of the target object in a target action posture according to the body posture parameters, the skin weights of the bones to the vertices, and the UV unfolding map.

[0132] For example, Figure 7 As shown, the body posture parameters of each body part can be filled into the UV expansion map of the corresponding body part through the body feature encoder to obtain the UV expansion map to be processed corresponding to each body part, and then the skin weight of each bone to the vertex is filled into these UV expansion maps to be processed to convert them into masks. For example, the mask is 1 where there is a weight, and the mask is 0 where there is no weight, so as to obtain the multi-layer feature values ​​corresponding to each body part. After that, the convolutional neural network in the body feature encoder can be used to encode the multi-layer feature values ​​corresponding to each body part to obtain the posture feature map corresponding to each body part. These posture features Figure 1 It can generally be represented by a two-dimensional matrix. Finally, by superimposing the posture feature maps of these body parts, that is, the two-dimensional matrix, a three-dimensional feature matrix can be obtained. The three-dimensional feature matrix is ​​the posture feature matrix of the target object in the target action posture, see Figure 7 .

[0133] The specific method of superimposing the posture feature graph can be found in the previous embodiment and will not be described in detail here.

[0134] 204. The electronic device performs feature encoding on the vertex position map through a local feature encoder to obtain a local vertex feature matrix.

[0135] For example, Figure 7 As shown, the vertex position map can be input into a local feature encoder, which encodes the vertex position map to obtain a feature matrix, such as a feature matrix of size 64x64x32, which is the local vertex feature matrix.

[0136] Among them, the local feature encoder can be implemented by using a convolutional neural network. The number of layers and network parameters of the convolutional neural network can be determined according to the needs of actual applications and are not limited here.

[0137] 205. The electronic device performs feature encoding on the vertex position graph through a global feature encoder to obtain a global vertex feature matrix.

[0138] For example, Figure 3 As shown, the vertex position map can be input into the global feature encoder, and the global feature encoder encodes the vertex position map into a feature vector, for example, a feature vector of size 1x1x2048, and then decodes the feature vector into a feature matrix, for example, a feature matrix of size 64x64x32, which is the global vertex feature matrix.

[0139] Similar to the local feature decoder, the global feature encoder can also be implemented using a convolutional neural network. The number of layers and network parameters of the convolutional neural network can be determined according to the needs of the actual application. However, when setting up, the number of network layers of the global feature decoder should be more than that of the local feature decoder, so that in addition to the differences in the learned features of the two feature decoders, the global feature encoder can pay more attention to global information.

[0140] It should be noted that steps 203 , 204 and 205 may be performed in any order.

[0141] 206. The electronic device merges the posture feature matrix, the local vertex feature matrix, and the global vertex feature matrix to obtain a merged feature matrix.

[0142] For example, if the size of the posture feature matrix, local vertex feature matrix and global vertex feature matrix are all 64x64x32, then Figure 7 As shown, the posture feature matrix, the local vertex feature matrix, and the global vertex feature matrix can be stacked in one direction to form a large three-dimensional matrix, such as a three-dimensional matrix of size 64x64x96, and the large three-dimensional matrix is ​​the merged feature matrix.

[0143] 207 . The electronic device obtains a plurality of two-dimensional facial images (ie, expression drive signals) of the target object, and then executes step 208 .

[0144] For example, taking the target object as Zhang San, multiple two-dimensional facial images of Zhang San can be obtained at this time, such as by using a camera or video camera to collect facial images containing various smiling expressions of Zhang San from multiple angles.

[0145] Steps 207 and 201 may be performed in any order.

[0146] 208. The electronic device performs feature encoding on these two-dimensional facial images through an expression encoder to obtain encoded facial feature vectors.

[0147] For example, these two-dimensional facial images can be encoded by an expression encoder to obtain vectors corresponding to multiple two-dimensional facial images. Then, the vectors corresponding to the multiple two-dimensional facial images are randomly weighted averaged to obtain the encoded facial feature vector. For details, please refer to the previous embodiments, which will not be described here.

[0148] 209 . The electronic device fuses the merged feature matrix and the encoded facial feature vector to obtain a fused signal, and then executes step 210 .

[0149] For example, see Figure 7 Specifically, the encoded facial feature vector can be processed by an FC and a small convolution decoder to convert it into a facial feature matrix (three-dimensional matrix). Then, the data used to represent facial movements in the merged feature matrix is ​​replaced with the data in the facial feature matrix (i.e., the facial feature matrix is ​​injected) to obtain the fused signal.

[0150] For example, the data in the upper left part of the merged feature matrix is ​​generally used to represent facial action data. Therefore, at this time, the fused signal can be obtained by simply replacing the data in the facial feature matrix with the data in the upper left part of the merged feature matrix. Figure 7 .

[0151] 210. The electronic device extracts Gaussian parameters from the fused signal through a Gaussian parameter decoder to obtain Gaussian parameter variation; for example, Figure 7 As shown, the details are as follows:

[0152] The fused signal is passed to the Gaussian parameter decoder, where the Gaussian parameter decoder is a convolution decoder. Through the Gaussian parameter decoder, the fused signal can be converted from a three-dimensional matrix to a non-vector representation, thereby converting the original three-dimensional matrix output result into a Gaussian parameter change, such as position change parameters, rotation change parameters, and scaling change parameters, etc. Of course, other Gaussian parameters can also be extracted at this time, such as Gaussian opacity parameters and Gaussian color parameters, etc.

[0153] 211. The electronic device obtains Gaussian parameters of a preset standard object in a standard action posture from the skeletal animation model.

[0154] For example, see Figure 7 Taking the standard action posture as a T-shaped posture as an example, at this time, the Gaussian parameters of the standard object under the T-shaped posture can be obtained from the skeletal animation model, such as basic position parameters, basic rotation parameters, and basic scaling parameters.

[0155] 212. The electronic device calculates the Gaussian parameters of the target object in the target action posture according to the Gaussian parameter variation and the Gaussian parameters of the standard object in the standard action posture. For example, the specific calculation may be as follows:

[0156] First, the Gaussian parameters of the standard object in the standard action posture are adjusted according to the Gaussian parameter variation to obtain the Gaussian parameters of the target object in the standard action posture; for example, see Figure 7 , if the target object is Zhang San and the standard action posture is "T-shaped posture", then the Gaussian parameters of Zhang San in the T-shaped posture can be obtained.

[0157] Secondly, the linear mixed skinning algorithm can be used to calculate the Gaussian parameters of the target object in the target action posture, such as the position parameters, rotation parameters and scaling parameters of the target object in the target action posture, based on the previously obtained body posture parameters and the Gaussian parameters of the target object in the standard action posture. For example, if the target object is Zhang San, and the target action posture is "standing and raising the left hand", then refer to Figure 7 , at this time, we can get the Gaussian parameters of Zhang San in the posture of "standing and raising his left hand".

[0158] The formula for calculating the Gaussian parameters (such as) of the target object under the target action posture can be as follows:

[0159]

[0160] Among them, Δμ is the position change parameter, Δq is the rotation change parameter, and Δs is the scaling change parameter. is the basic position parameter, is the basic rotation parameter, is the basic scaling parameter, μ p is the position parameter of the target object in the target action posture, q p is the rotation parameter of the target object in the target action posture, s is the scaling parameter of the target object in the target action posture; LBS represents the linear blending skinning algorithm, θ is the body posture parameter, and W is the center of gravity interpolation skinning weight of the skeletal animation model, which can be obtained from the skeletal animation model.

[0161] 213. The electronic device renders a two-dimensional image of the target object in the target action posture according to the Gaussian parameters of the target object in the target action posture, and obtains a digital human corresponding to the target object.

[0162] After that, the digital person corresponding to the target object can be displayed on the electronic screen.

[0163] As can be seen from the above, after acquiring the two-dimensional facial image and body posture parameters of the target object, this embodiment can fuse them. Then, on the one hand, Gaussian parameters are extracted from the fused signals to obtain Gaussian parameter changes. On the other hand, Gaussian parameters of the preset standard object in the standard action posture are obtained from the skeletal animation model. Then, according to the Gaussian parameter changes and the Gaussian parameters of the standard object in the standard action posture, the Gaussian parameters of the target object in the target action posture can be calculated, and the two-dimensional image of the target object in the target action posture is rendered accordingly, so that the digital human corresponding to the target object can be displayed on the screen; since this solution only needs to input the two-dimensional facial image and limb driving signal of the target object to render the 3D Gaussian digital human corresponding to the target object, and there is no need to determine the grid shape and render it separately for each posture and expression of the target object, therefore, the amount of calculation and the number of renderings can be greatly reduced, and the generation efficiency can be improved. Moreover, since it is rendered based on Gaussian parameters, compared with the technology based on grid rendering, it can add more rendering details and greatly improve the rendering quality.

[0164] In addition, since this solution can quickly and easily generate 3D Gaussian digital humans with a large number of body movements and rich facial expressions, it can create conditions for the application of digital human technology in extended reality and telepresentation, greatly enhancing its applicability.

[0165] In order to better implement the above method, an embodiment of the present application also provides a digital human generation device, which can be integrated into an electronic device, such as a server or a terminal, and the terminal may include a tablet computer, a laptop computer and / or a personal computer, etc.

[0166] like Figure 8 As shown, the digital human generation device may include a creation unit 301, an acquisition unit 302, a fusion unit 303, an extraction unit 304, a calculation unit 305 and a rendering unit 306, as follows:

[0167] (1) Establishing unit 301;

[0168] The building unit 301 is used to build a three-dimensional model of the target object in a target action posture through a skeletal animation model.

[0169] (2) Acquisition unit 302;

[0170] The acquisition unit 302 is used to acquire the expression drive signal of the target object and acquire the limb drive signal of the target object in the target action posture from the three-dimensional model.

[0171] Among them, the expression driving signal can be a traditional 3DMM or a two-dimensional facial image; the limb driving signal can be a body posture parameter, or other parameters that can represent limb movements. For details, please refer to the previous method embodiment, which will not be repeated here.

[0172] (3) Fusion unit 303

[0173] The fusion unit 303 is used to fuse the expression drive signal and the limb drive signal to obtain a fused signal.

[0174] For example, the fusion unit 303 may be specifically configured to fuse the expression drive signal and the body posture parameter using an encoder in a preset digital human generation network to obtain a fused signal.

[0175] The digital human generation network may include an encoder and a Gaussian parameter decoder. Optionally, the encoder may be set according to the needs of the actual application, for example, the encoder includes a body feature encoder, a local feature encoder, a global feature encoder, and an expression encoder, etc. For details, please refer to the previous method embodiment, which will not be described in detail here.

[0176] (4) Extraction unit 304;

[0177] The extraction unit 304 is used to extract Gaussian parameters from the fused signal to obtain Gaussian parameter variation.

[0178] For example, the extraction unit 304 can be specifically used to extract Gaussian parameters from the fused signal by presetting a Gaussian parameter decoder in the digital human generation network to obtain the Gaussian parameter variation.

[0179] (5) a calculation unit 305;

[0180] The calculation unit 305 is used to obtain the Gaussian parameters of the preset standard object in the standard action posture from the skeletal animation model, and calculate the Gaussian parameters of the target object in the target action posture according to the Gaussian parameter change and the Gaussian parameters of the standard object in the standard action posture. The specific calculation method can be found in the previous method embodiment and will not be repeated here.

[0181] (6) rendering unit 306;

[0182] The rendering unit 306 is used to render a two-dimensional image of the target object in the target action posture according to the Gaussian parameters of the target object in the target action posture, so as to obtain a digital human corresponding to the target object.

[0183] In specific implementation, the above units can be implemented as independent entities, or can be arbitrarily combined to be implemented as the same or several entities. The specific implementation of the above units can refer to the previous method embodiments, which will not be repeated here.

[0184] As can be seen from the above, after the digital human generation device of this embodiment obtains the expression drive signal and the limb drive signal of the target object, the fusion unit 303 can fuse the expression drive signal and the limb drive signal, and then the extraction unit 304 extracts the Gaussian parameters from the fused signals to obtain the Gaussian parameter variation. Then, the calculation unit 305 obtains the Gaussian parameters of the preset standard object in the standard action posture from the skeletal animation model, and calculates the Gaussian parameters of the target object in the target action posture according to the Gaussian parameter variation and the Gaussian parameters of the standard object in the standard action posture, and then the rendering unit 305 performs the operation of the rendering unit 306. The rendering unit 306 renders a two-dimensional image of the target object in the target action posture accordingly, and obtains a digital human corresponding to the target object; since this solution only needs to input the expression driving signal and the limb driving signal of the target object to render the digital human corresponding to the target object, without having to determine the grid shape and render it separately for each posture and each expression of the target object, it can greatly reduce the amount of calculation and the number of renderings, and improve the generation efficiency. Moreover, since it is rendered based on Gaussian parameters, it can add more rendering details and greatly improve the rendering quality compared to the technology based on grid rendering.

[0185] Correspondingly, the embodiment of the present application also provides an electronic device that can integrate any digital human generation device provided in the embodiment of the present application. The specific details of the digital human generation device can be found in the previous embodiments and will not be described here.

[0186] A person of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0187] To this end, the embodiment of the present application further provides a storage medium, in which a plurality of instructions are stored, and the instructions can be loaded by a processor to execute the steps in any one of the digital human generation methods provided in the embodiment of the present application. For example, the instructions can execute the following steps:

[0188] A three-dimensional model of a target object in a target action posture is established through a skeletal animation model; an expression driving signal of the target object is obtained, and a limb driving signal of the target object in the target action posture is obtained from the three-dimensional model; the expression driving signal and the limb driving signal are fused to obtain a fused signal; Gaussian parameters are extracted from the fused signal to obtain a Gaussian parameter variation; Gaussian parameters of a preset standard object in a standard action posture are obtained from the skeletal animation model; Gaussian parameters of the target object in the target action posture are calculated according to the Gaussian parameter variation and the Gaussian parameters of the standard object in the standard action posture; a two-dimensional image of the target object in the target action posture is rendered according to the Gaussian parameters of the target object in the target action posture, and a digital human corresponding to the target object is obtained.

[0189] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0190] The storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0191] Since the instructions stored in the storage medium can execute the steps in any digital human generation method provided in the embodiments of the present application, the beneficial effects that can be achieved by any digital human generation method provided in the embodiments of the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0192] The above is a detailed introduction to a digital human generation method, device and storage medium provided in the embodiments of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A method for generating a digital human, characterized in that: include: Establish a three-dimensional model of the target object in the target action posture through the skeletal animation model; Acquire an expression driving signal of the target object, and acquire a limb driving signal of the target object in a target action posture from the three-dimensional model; fusing the expression drive signal and the limb drive signal to obtain a fused signal; Extracting Gaussian parameters from the fused signal to obtain Gaussian parameter variation; Obtaining Gaussian parameters of a preset standard object in a standard action posture from the skeletal animation model; Calculating the Gaussian parameters of the target object in the target action posture according to the Gaussian parameter variation and the Gaussian parameters of the standard object in the standard action posture; According to the Gaussian parameters of the target object in the target action posture, a two-dimensional image of the target object in the target action posture is rendered to obtain a digital human corresponding to the target object.

2. The method according to claim 1, characterized in that The limb drive signal is a body posture parameter, and obtaining the limb drive signal of the target object in the target action posture from the three-dimensional model includes: Obtaining body posture parameters of each body part of the target object in a target action posture from the three-dimensional model; The expression drive signal and the limb drive signal are fused to obtain a fused signal, specifically: an encoder in a preset digital human generation network is used to fuse the expression drive signal and the body posture parameter to obtain a fused signal; The extracting Gaussian parameters from the fused signal to obtain the Gaussian parameter variation is specifically: extracting Gaussian parameters from the fused signal by presetting a Gaussian parameter decoder in the digital human generation network to obtain the Gaussian parameter variation.

3. The method according to claim 2, characterized in that The encoder in the digital human generation network includes a body feature encoder, a local feature encoder, a global feature encoder and an expression encoder. The encoder in the preset digital human generation network is used to fuse the expression drive signal and the limb drive signal to obtain a fused signal, including: From the three-dimensional model, obtaining a UV unfolding diagram of each body part of the target object in a target action posture, a skinning weight of a bone to a vertex, and a vertex position diagram of the three-dimensional model; Using a body feature encoder, a posture feature matrix of the target object in a target action posture is generated according to the body posture parameters, the skin weights of the bones to the vertices, and the UV unfolding map; Performing feature encoding on the vertex position map by a local feature encoder to obtain a local vertex feature matrix; Performing feature encoding on the vertex position graph by a global feature encoder to obtain a global vertex feature matrix; Merging the posture feature matrix, the local vertex feature matrix, and the global vertex feature matrix to obtain a merged feature matrix; Performing feature encoding on the expression drive signal through an expression encoder to obtain an encoded facial feature vector; The merged feature matrix and the encoded facial feature vector are fused to obtain a fused signal.

4. The method according to claim 3, characterized in that The step of fusing the merged feature matrix and the encoded facial feature vector to obtain a fused signal includes: Converting the encoded facial feature vector into a three-dimensional matrix to obtain a facial feature matrix; The data used to represent facial movements in the merged feature matrix are replaced with the data in the facial feature matrix to obtain a fused signal.

5. The method according to claim 3, characterized in that: The method of using a body feature encoder to generate a posture feature matrix of the target object in a target action posture according to the body posture parameters, the skin weights of the bones to the vertices, and the UV unfolding map includes: Through the body feature encoder, the body posture parameters of each body part, the skinning weight of the bone to the vertex, and the UV unfolding map are encoded to obtain the posture feature maps of multiple body parts; Superimposing the posture feature maps of the multiple body parts to obtain a posture feature matrix of the target object in the target action posture; The method encodes the body posture parameters of each body part, the skin weights of the bones to the vertices, and the UV unfolding map through the body feature encoder to obtain posture feature maps of multiple body parts, including: Through the body feature encoder, the body posture parameters of each body part are respectively filled into the UV unfolding map of the corresponding body part to obtain the UV unfolding map to be processed corresponding to each body part; The UV unfolded image to be processed is converted into a mask according to the skinning weights of the bones to the vertices, so as to obtain multi-layer feature values ​​corresponding to each body part; The multi-layer feature values ​​corresponding to each body part are encoded respectively to obtain posture feature maps of multiple body parts.

6. The method according to claim 3, characterized in that Obtaining the vertex position map of the three-dimensional model includes: Acquire information of each vertex of the three-dimensional model of the target object in the target action posture; The information of each vertex of the three-dimensional model is associated with the corresponding vertex in the UV unfolding diagram to obtain a vertex position diagram of the three-dimensional model.

7. The method according to any one of claims 2 to 6, characterized in that: The step of calculating the Gaussian parameters of the target object in the target action posture according to the Gaussian parameter variation and the Gaussian parameters of the standard object in the standard action posture comprises: Adjusting the Gaussian parameters of the standard object in the standard action posture according to the Gaussian parameter variation to obtain the Gaussian parameters of the target object in the standard action posture; According to the body posture parameters and the Gaussian parameters of the target object in the standard action posture, a linear mixed skinning algorithm is used to calculate the Gaussian parameters of the target object in the target action posture.

8. The method according to any one of claims 2 to 6, characterized in that: Before fusing the expression drive signal and the limb drive signal using the encoder in the preset digital human generation network to obtain the fused signal, the method further includes: Collect two-dimensional images corresponding to multiple training samples to obtain real images; A three-dimensional model of the training sample in the target action posture is established through the skeletal animation model to obtain the three-dimensional model of the training sample; Acquire the expression driving signal of the training sample, and acquire the limb driving signal of the training sample in the target action posture from the three-dimensional model of the training sample; The encoder in the preset convolutional neural network is used to fuse the expression drive signal and the limb drive signal of the training sample to obtain the fused signal of the training sample; The Gaussian parameter decoder in the convolutional neural network is used to extract the Gaussian parameter from the fused signal of the training sample to obtain the Gaussian parameter variation of the training sample; Obtaining Gaussian parameters of a preset standard object in a standard action posture from the skeletal animation model; Calculate the Gaussian parameters of the training sample in the target action posture according to the Gaussian parameter variation of the training sample and the Gaussian parameters of the standard object in the standard action posture; Rendering a two-dimensional image of the training sample in the target action posture according to the Gaussian parameters of the training sample in the target action posture to obtain a predicted image; The preset loss function is used to converge the convolutional neural network according to the real image and the predicted image to obtain a digital human generation network.

9. A digital human generation device, characterized in that: include: A building unit, used for building a three-dimensional model of a target object in a target action posture through a skeletal animation model; An acquisition unit, used to acquire an expression drive signal of a target object, and acquire a limb drive signal of the target object in a target action posture from the three-dimensional model; A fusion unit, used for fusing the expression drive signal and the limb drive signal to obtain a fused signal; An extraction unit, used for extracting Gaussian parameters from the fused signal to obtain Gaussian parameter variation; A calculation unit, used for obtaining Gaussian parameters of a preset standard object in a standard action posture from the skeletal animation model, and calculating Gaussian parameters of the target object in a target action posture according to the Gaussian parameter variation and the Gaussian parameters of the standard object in the standard action posture; The rendering unit is used to render a two-dimensional image of the target object in the target action posture according to the Gaussian parameters of the target object in the target action posture, so as to obtain a digital human corresponding to the target object.

10. A storage medium, characterized in that: The storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor to execute the steps in the method for generating a digital human as claimed in any one of claims 1 to 8.

Images