Method and device for generating 3D digital human action sequence, electronic equipment and storage medium

Abstract

The invention provides a 3D digital human action sequence generation method and device, electronic equipment and a storage medium. Receiving first multi-modal data of the 3D digital human, wherein the first multi-modal data comprises voice data; inputting the first multi-modal data into a trained limb action generation model to generate an action sequence of the 3D digital human; wherein the training set of the limb action generation model comprises a first multi-modal data sample and a corresponding action sequence sample. Multi-modal data including voice data is input into a limb action generation model to generate an action sequence of a 3D digital human. Compared with the prior art that the action sequence of the 3D digital human is generated by using a voice animation synthesis technology, the quality of the generated action sequence is better, and the driven 3D digital human is better in the aspects of whole body coordination, action fluency, action rationality, figure attractiveness and the like.

Description

Technical Field

[0001] The present invention relates to the technical field of 3D motion generation, and particularly to a method, device, electronic device, and storage medium for generating a 3D digital human motion sequence. Background Art

[0002] A 3D virtual digital human, abbreviated as a "3D digital human", is a digital human image created using digital technology and similar to a real human image. It has human characteristics such as appearance, gender, and personality, and has expression capabilities such as speech, facial expressions, and body movements. It can be understood that the external image and expression capabilities of a 3D digital human should be as close to real humans as possible. Considering that real humans are naturally coordinated in expressions such as speech, facial expressions, and body movements, a 3D digital human should also be coordinated in expressions such as speech, facial expressions, and body movements to make the 3D digital human closer to real humans as a whole.

[0003] For the motion sequence used to drive a 3D digital human to perform actions, the prior art can already generate a motion sequence by inputting speech data into a STA (Speech To Animation) model. However, the quality of the motion sequence generated by the STA model in the prior art is poor, and the gap with the real human motion sequence is large, resulting in a large gap between the actions of the 3D digital human and those of real humans.

[0004] Therefore, in order to make the actions of the 3D digital human closer to real humans, it is necessary to consider the problem of improving the quality of the motion sequence. Summary of the Invention

[0005] The present invention provides a method, device, electronic device, and storage medium for generating a 3D digital human motion sequence to solve the technical problem of poor quality of the motion sequence of the 3D digital human in the prior art.

[0006] The present invention provides a method for generating a 3D digital human motion sequence, including:

[0007] Receiving first multi-modal data of a 3D digital human, where the first multi-modal data includes speech data;

[0008] Inputting the first multi-modal data into a trained limb motion generation model to generate a motion sequence of the 3D digital human;

[0009] Wherein, the training set of the limb motion generation model includes first multi-modal data samples and corresponding motion sequence samples.

[0010] According to the method for generating a 3D digital human motion sequence provided by the present invention, the first multi-modal data further includes at least one of key motion data, style type data, and lower body walking motion sequence.

[0011] A method for generating a 3D digital human action sequence provided by the present invention, wherein the limb action generation model is a diffusion model;

[0012] The step of inputting the first multi-modal data into the trained limb action generation model to generate the action sequence of the 3D digital human includes:

[0013] Generating Gaussian noise data;

[0014] Using the first multi-modal data as a condition for the diffusion model to perform denoising processing on the Gaussian noise data to generate the action sequence of the 3D digital human.

[0015] A method for generating a 3D digital human action sequence provided by the present invention, wherein the first multi-modal data includes a lower body walking action sequence, the limb action generation model includes a body action generation model and a hand action generation model, and the action sequence includes a body action sequence and a hand action sequence;

[0016] The step of inputting the first multi-modal data into the trained limb action generation model to generate the action sequence of the 3D digital human includes:

[0017] Inputting the first multi-modal data into the body action generation model to generate the body action sequence;

[0018] Replacing the lower body walking action sequence in the first multi-modal data with the body action sequence to obtain second multi-modal data;

[0019] Inputting the second multi-modal data into the hand action generation model to generate the hand action sequence;

[0020] Wherein, the training set of the body action generation model includes the first multi-modal data sample and the corresponding body action sequence sample, and the training set of the hand action generation model includes the second multi-modal data sample and the corresponding hand action sequence sample.

[0021] A method for generating a 3D digital human action sequence provided by the present invention, wherein the diffusion model is a UNet network architecture, and the UNet network is composed of a plurality of encoding blocks and decoding blocks, and each encoding block or decoding block includes a convolutional neural network and a Transformer neural network.

[0022] A method for generating a 3D digital human action sequence provided by the present invention, wherein the step of inputting the first multi-modal data into the trained limb action generation model includes:

[0023] Using a speech recognition model to extract the speech features of the speech data;

[0024] Input the speech features into the body motion generation model.

[0025] According to a method for generating a 3D digital human motion sequence provided by the present invention, before receiving the first multi-modal data of the 3D digital human, the method further includes:

[0026] Receive the line text;

[0027] Use speech synthesis technology to convert the line text into the speech data.

[0028] The present invention also provides a 3D digital human driving method, including:

[0029] Apply any one of the above methods for generating a 3D digital human motion sequence to generate a 3D digital human motion sequence;

[0030] Perform redirection processing on the 3D digital human motion sequence;

[0031] Use the redirected 3D digital human motion sequence to drive the 3D digital human.

[0032] The present invention also provides a 3D digital human video generation method, including:

[0033] Apply the 3D digital human driving method above to drive the 3D digital human;

[0034] Render the 3D digital human to generate a 3D digital human video.

[0035] The present invention also provides a device for generating a 3D digital human motion sequence, including:

[0036] A receiving module, configured to receive the first multi-modal data of the 3D digital human, where the first multi-modal data includes speech data;

[0037] A generating module, configured to input the first multi-modal data into a trained body motion generation model to generate the motion sequence of the 3D digital human;

[0038] Wherein, the training set of the body motion generation model includes first multi-modal data samples and corresponding motion sequence samples.

[0039] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, where when the processor executes the program, it implements any one of the above methods for generating a 3D digital human motion sequence, a 3D digital human driving method, or a 3D digital human video generation method.

[0040] The present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the generation method, driving method, or video generation method of the 3D digital human action sequence as described in any one of the above is implemented.

[0041] The present invention also provides a computer program product, including a computer program. When the computer program is executed by a processor, the generation method, driving method, or video generation method of the 3D digital human action sequence as described in any one of the above is implemented.

[0042] The generation method, device, electronic device, and storage medium of the 3D digital human action sequence provided by the present invention input multi-modal data including voice data into a limb action generation model to generate an action sequence of the 3D digital human. Compared with the prior art that uses speech-to-animation (STA) technology to generate the action sequence of the 3D digital human, the generated action sequence has better quality, and the driven 3D digital human performs better in multiple aspects such as whole-body coordination, action fluency, action rationality, and character aesthetics. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0044] Figure 1 is a schematic diagram of a frame of action data in the 3D digital human action sequence.

[0045] Figure 2 is a flowchart of the generation method of the 3D digital human action sequence provided by the present invention.

[0046] Figure 3 is a schematic diagram of the principle of forward diffusion of the diffusion model provided by the present invention.

[0047] Figure 4 is a schematic diagram of the principle of reverse diffusion of the diffusion model provided by the present invention.

[0048] Figure 5 is a flowchart of step S2 provided by the present invention.

[0049] Figure 6 is a schematic diagram of generating a body action sequence provided by the present invention.

[0050] Figure 7It is a schematic diagram of generating a hand motion sequence provided by the present invention.

[0051] Figure 8 It is a schematic structural diagram of the UNet network architecture provided by the present invention.

[0052] Figure 9 It is a schematic flowchart of a 3D digital human driving method provided by the present invention.

[0053] Figure 10 It is a schematic flowchart of a 3D digital human video generation method provided by the present invention.

[0054] Figure 11 It is a schematic structural diagram of a device for generating a 3D digital human motion sequence provided by the present invention.

[0055] Figure 12 It is a schematic structural diagram of an electronic device provided by the present invention. Detailed implementation manners

[0056] To make the objectives, technical solutions and advantages of the present invention clearer, the technical solutions in the present invention will be clearly and completely described below with reference to the accompanying drawings in the present invention. Obviously, the described embodiments are some but not all of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present invention without making creative efforts fall within the scope of protection of the present invention.

[0057] It should be noted that in the description of the present invention, the terms "include", "comprise" or any other variant thereof are intended to cover a non-exclusive inclusion, such that a process, method, article or device including a series of elements not only includes those elements but also includes other elements not expressly listed, or further includes elements inherent to such process, method, article or device. Without further limitation, an element defined by the phrase "including a..." does not exclude the existence of additional identical elements in the process, method, article or device including the element. The orientation or positional relationship indicated by terms such as "upper", "lower", etc. is based on the orientation or positional relationship shown in the drawings, and is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and thus cannot be construed as a limitation of the present invention. Unless otherwise clearly specified and defined, the terms "mount", "connect", "couple" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be directly connected or indirectly connected through an intermediate medium, and it can be the internal communication of two elements. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0058] The terms "first", "second", etc. in the present invention are used to distinguish similar objects, rather than to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present invention can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of the same type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" means at least one of the connected objects, and the character " / " generally indicates an "or" relationship between the associated objects before and after.

[0059] A 3D virtual digital human (abbreviated as "3D digital human") refers to a digital human image created by using digital technology and similar to the human image, with specific personal characteristics such as appearance, gender, and personality, and the ability to express through voice, facial expressions, and body movements. Further, 3D digital humans can be divided into interactive 3D digital humans and non-interactive 3D digital humans. Interactive 3D digital humans also have the ability to recognize the external environment and communicate and interact with people.

[0060] Simply put, a 3D digital human requires at least a specific appearance as the external image, as well as expression abilities such as voice, facial expressions, and body movements, and the closer it is to a real human in terms of external image and expression abilities, the better. It can be understood that real humans are naturally consistent in expressions such as voice, facial expressions, and body movements. This consistency includes both the smoothness and coherence of the voice, the naturalness of the facial expressions, and the coordination of the body movements, as well as the overall consistency of the voice, facial expressions, and body movements.

[0061] In related industries, the action sequences of real humans are usually collected through a full-body motion capture system and recorded in the form of the three-dimensional spatial coordinates of the key points of the full-body movements. By inputting these action sequences into software, any 3D digital human character model can be driven to make the same actions as real humans through a redirection algorithm. That is to say, as long as the action sequences of real humans are continuously collected, 3D digital humans can make various actions just like real humans.

[0062] One frame of action data in the 3D digital human action sequence is as Figure 1As shown in the figure. Among them, the action sequence refers to a set composed of several frames of action data arranged in chronological order, which is used to drive the 3D digital human to simulate the movements and behaviors of real humans. It should be noted that in the video world, each second of transformation consists of several pictures, which are called "frames", and the number of frames per second is called the "frame rate". Each frame of picture is a static image. Displaying the frames quickly and continuously creates the illusion of motion. Therefore, animations with higher frame rates are smoother and more realistic. In scenarios with higher requirements for picture smoothness, such as live sports events and e-sports video production, frame rates of 50 frames per second or 60 frames per second are used. It can be seen that the continuous pictures in the video are actually composed of several static pictures. Similar to the principle of video, the action sequence used to drive the 3D digital human to simulate human movements and behaviors is also composed of frame-by-frame action data arranged in chronological order.

[0063] Figure 1 As shown in the figure, a frame of action data contains the positions of the whole-body bone points in space. Hundreds of consecutive frames of action data characterize the position changes of the whole-body bone points within the corresponding time period. Using these hundreds of consecutive frames of action data to drive the 3D digital human can make the 3D digital human continuously perform corresponding actions within the corresponding time period.

[0064] However, the above scheme is costly. It not only requires the use of expensive motion capture equipment but also requires professional motion capture actors to perform various actions to achieve the acquisition of the action sequence.

[0065] With the continuous development of computer software technology, it has been possible to automatically generate action sequences through computer software to drive the 3D digital human character model to perform corresponding limb actions. Although some actions are still relatively rigid, they can initially meet the usage requirements.

[0066] Based on the foregoing description, it can be seen that the expressions of the 3D digital human include various forms such as speech, facial expressions, and limb actions. To make the expression ability of the 3D digital human close to that of real humans, not only do the limb actions need to be coordinated, but also different forms of expressions need to be consistent as a whole. For example, when the 3D digital human expresses happy or excited speech content, the expression and limb actions should also show happiness or excitement, which conforms to the expression habits of real humans.

[0067] In order to make the speech and body movements of 3D digital humans consistent, speech animation synthesis technology (Speech To Animation, STA) is adopted in related products to generate the action sequences of 3D digital humans, so as to achieve the consistency between the speech and body movements of 3D digital humans. Specifically, by inputting a piece of speech into the STA model, the corresponding action sequence can be automatically generated according to the content, tone, etc. of the speech, and the 3D digital human model can be driven to perform corresponding body movements while speaking. However, there is still a certain gap between the action sequences generated by the STA model and the action sequences directly collected from real humans, resulting in differences in the whole-body coordination, movement fluency, and rationality of the body movements of the driven 3D digital humans compared with real humans.

[0068] The following will Figures 2 - 12 describe the method, device, electronic device, and storage medium for generating the action sequence of the 3D digital human provided by the present invention.

[0069] Figure 2 is a schematic flowchart of the method for generating the action sequence of the 3D digital human provided by the present invention, as Figure 2 shown, including but not limited to the following steps:

[0070] Step S1, receiving the first multi-modal data of the 3D digital human, where the first multi-modal data includes speech data.

[0071] Among them, multi-modal data refers to a data set containing multiple types of data, such as various forms of images, texts, audios, and videos. The speech data exists in the form of audio. Therefore, the first multi-modal data includes at least two types of data.

[0072] In some embodiments, the speech data can be collected by an audio acquisition device. It can be understood that audio is a data that continuously changes over time, with temporal coherence and continuity. The audio corresponds to a time axis, and the length of the time axis determines the duration of the audio. Each time point on the time axis can be marked by a timestamp.

[0073] Step S2, inputting the first multi-modal data into the trained body movement generation model to generate the action sequence of the 3D digital human;

[0074] Among them, the training set of the body movement generation model includes the first multi-modal data sample and the corresponding action sequence sample.

[0075] In some embodiments, the limb movement generation model can be one of a Generative Adversarial Networks (GAN) model, a Transformer model, a diffusion model, a pre-trained model, and a Contrastive Language-Image Pre-Training (CLIP) multi-modal pre-trained neural network model.

[0076] Compared with the prior art that uses Speech To Animation (STA) technology to generate the action sequence of a 3D digital human, the present invention inputs multi-modal data including speech data into the trained limb movement generation model to generate the action sequence of the 3D digital human. The generated action sequence has better quality, and the driven 3D digital human performs better in multiple aspects such as whole-body coordination, action fluency, action rationality, and character aesthetics.

[0077] In some embodiments, the first multi-modal data of the present invention may further include at least one of key action data, style type data, and lower-body walking action sequence.

[0078] Among them, the speech data, key action data, and lower-body walking action sequence are all related to time. The speech data is continuous data, while the key action data and the lower-body walking action sequence are both discrete data. The style type data can correspond different style types to different discrete numbers in an encoded manner. For example, the key action data can be in the form of a matrix, where the number of columns of the matrix represents the number of frames corresponding to the key action (i.e., the duration of the key action), and the set of numbers in each column of the matrix represents one frame of key action data. The lower-body walking action sequence can also be in the form of a matrix, where the number of columns of the matrix represents the number of frames of the lower-body walking action sequence, and the set of numbers in each column of the matrix represents one frame of lower-body walking action data. The style type data can be in the form of a vector, and the combination of discrete numbers included in the vector corresponds to different style types.

[0079] Among them, the key action data is the data corresponding to the key actions that the user requires the 3D digital human to perform. For example, if the user requires the 3D digital human to perform a waving action at a specified time point (determined by the time stamp on the time axis), the key action data is the data corresponding to the specified time point and the waving action. If the first multi-modal data includes key action data, then step S2 can generate an action sequence that matches the key action data, further improving the quality of the action sequence. After driving the 3D digital human based on the action sequence, the 3D digital human can perform actions that are coordinated with the key actions.

[0080] Among them, the style type data characterizes the style of the generated action sequence, such as happy, lost, serious, lively, etc. The present invention allows the user to specify the style type for the action sequence to be generated and input it into the trained body movement generation model in the form of style type data for generating an action sequence of the specified style type. For example, if the user pre-specifies that the style type of the generated action sequence is happy, then the action sequence generated by the body movement generation model will drive the 3D digital human to make a large number of actions representing happiness. That is to say, if the first multi-modal data includes style type data, step S2 can generate an action sequence corresponding to the style type, enabling the user to specify the style type of the action sequence. After driving the 3D digital human based on the action sequence, the 3D digital human can make actions consistent with the style type specified by the user.

[0081] Among them, in some scenarios, the 3D digital human needs to walk along the walking route specified by the user. The lower body walking action sequence is a set of frame-by-frame data corresponding to the walking action. The lower body can be understood as the part below the waist. If the first multi-modal data includes the lower body walking action sequence, step S2 can generate an action sequence matching the lower body walking action sequence, enabling the user to specify the walking route of the action sequence. After driving the 3D digital human based on the action sequence, the 3D digital human can make actions coordinated with the walking action.

[0082] It can be understood that the first multi-modal data of the present invention contains at least two types of data. For the convenience of inputting into the body movement generation model, the present invention can pre-fuse various types of data to obtain fused data and input the fused data as a whole into the trained body movement generation model.

[0083] In some embodiments, the body movement generation model of the present invention can be a diffusion model;

[0084] Step S2 may include:

[0085] Generating Gaussian noise data;

[0086] Using the first multi-modal data as a condition for the diffusion model to denoise the Gaussian noise data to generate an action sequence of the 3D digital human.

[0087] The working mechanism of the diffusion model is divided into two stages: forward diffusion and reverse diffusion. Figure 3 is the forward diffusion stage. Figure 4 is the reverse diffusion stage. T is the total number of steps for adding noise or denoising, t is the digital label corresponding to the current step (the digital label corresponding to the noise addition steps is an integer sequence increasing from 0 to T, and the digital label corresponding to the denoising steps is an integer sequence decreasing from T to 0), x0 is the regular data, x T is the Gaussian noise data, x tis the data after adding noise at the t-th step.

[0088] As Figure 3 shown, forward diffusion is a process of gradually increasing the complexity of data. A simple sample is randomly drawn from a sample set that follows the target distribution. Through a series of reversible and gradual modifications, a certain amount of noise (the noise is structured and controllable) is introduced into the sample at each step. As the noise is continuously introduced, the complexity of the sample increases continuously, and finally a sample very similar to the required complex data distribution (such as Gaussian distribution) is obtained, thus transforming a data distribution into a noise distribution. That is, continuously introducing noise into the regular data x0 to obtain Gaussian noise x T .

[0089] As Figure 4 shown, reverse diffusion is the process of generating regular data x0, which can recover the regular data x0 from the Gaussian noise x T . First, Gaussian noise x T is randomly generated, and then the operations opposite to those of forward diffusion are gradually executed to gradually remove the noise, thereby recovering the regular data x0.

[0090] It should be noted that the process of gradually removing noise in reverse diffusion can be defined as reverse sampling, and reverse sampling is implemented by a reverse sampler. Therefore, the key is to construct a reverse sampler that meets the requirements. The reverse sampler includes denoising diffusion probabilistic models (DDPM) and denoising diffusion implicit models (DDIM). Among them, the DDPM sampler defines a random mapping, while the DDIM sampler defines a deterministic mapping. Specifically, using the DDPM sampler to denoise x T , the output x0 is almost independent of the input x T , x t does not depend on x T , but only depends on the previous step of x t-1 , forming a reverse Markov chain. And using the DDIM sampler to denoise x T , the output x0 strongly depends on the input x T . For a given x T , the output x0 has a high degree of consistency, and the entire denoising process is a non-Markov process.

[0091] It is understandable that in the Markov process, the data at adjacent times has a very tight binding relationship. Therefore, denoising needs to be carried out step by step. In the non-Markov process, skip-step denoising can be achieved, that is, multiple steps are skipped each time, and the data generation is accelerated by reducing the number of steps. For example, when the value of T is 1000, the DDPM sampler needs to perform 1000 denoising processes to generate x0 based on the input x T while the DDIM sampler can skip 5 steps each time, and only 200 denoising processes are needed to achieve the same processing effect.

[0092] The training process of the diffusion model includes: adding random noise to x t-1 at the t-th step to obtain the noisy data x t , recording the random noise at the t-th step as the training data for the corresponding denoising step; at the (T - t)-th step of denoising, based on the conditional sample corresponding to the regular data sample x0, the noisy data x t and the value of (T - t), predicting the random noise introduced at the t-th step, calculating the loss between the predicted noise and the actual random noise introduced at the t-th step, adjusting the parameters of the diffusion model based on the loss calculated for each denoising step, and performing denoising and calculating the loss again. Repeat the above steps multiple times until the denoising of the diffusion model converges.

[0093] Flow Matching is a new training paradigm for diffusion models. First, it is assumed that there is a certain mapping relationship between x T and x0, and the transformation process from x T to x0 is continuous and differentiable. Then, there is at least one link (i.e., Flow) in the transformation process from x T to x0. It is understandable that the transformation process from x T to x0 is a dynamic movement process. Therefore, there is also a velocity vector v t at the t-th step, which is used to characterize the dynamic change at the t-th step. That is to say, the transformation from x T to x0 can be regarded as the superposition of the corresponding v t at each step acting on x T to generate x0. It can be seen that as long as the corresponding v t at each step is determined, the transformation from x T to x0 can be completed. Specifically, it can be predicted through a neural network model, predicting the velocity vector v t at the t-th step based on the data x t at the t-th step.

[0094] In the training stage of the diffusion model, as x0 is the training sample, it is easy to determine the data x at the t-th stept , and then calculate the corresponding velocity vector v t , and optimize the above neural network model through the loss function to continuously improve the prediction effect of the neural network model. Repeatedly training the model with different x0 can optimize the neural network model in the direction of being balanced in all velocity angles.

[0095] In the usage stage of the diffusion model, set the step size to △t. Then, the corresponding time for each processing is △t, 2△t,..., 1, and x T After steps, x0 can be obtained.

[0096] It should be noted that there are actually multiple links between x T and x0. Although the conversion can be finally completed along different links, there is an optimal transport path (Optimal Transport, OT). Along this path, x T can move to x0 along a straight line at a constant speed, and the path is simple and direct.

[0097] Compared with the traditional training method of the diffusion model, using Flow Matching to train the diffusion model can find the optimal transport path from multiple paths, so that in the model usage stage, the Gaussian noise data can be converted into regular data in the most direct and effective way.

[0098] It should be noted that the limb movement generation model of the present invention is a diffusion model. The input of the model is Gaussian noise data, the condition is the first multi-modal data, and the output is the action sequence of the 3D digital human. Therefore, when training the model, the training data set is the first multi-modal data sample and the corresponding action sequence sample. The data x t-1 and x t are respectively obtained after adding noise to the action sequence sample for t - 1 steps and t steps. When adding noise at the t-th step, random noise is introduced for x t-1 to obtain the noise-added data x t . Record the random noise at the t-th step as the training data for the corresponding denoising step; when denoising at the (T - t)-th step, based on the first multi-modal data sample, the noise-added data x t and the value of (T - t), predict the random noise introduced at the t-th step, calculate the loss between the predicted noise and the random noise actually introduced at the t-th step. Based on the loss calculated for each denoising step, adjust the parameters of the diffusion model, and perform denoising and loss calculation again. Repeat the above steps multiple times until the diffusion model converges in denoising, that is, the training of the model is completed.

[0099] When the present invention uses a diffusion model to generate an action sequence of a 3D digital human, based on the trained diffusion model, using the first multi-modal data as a condition, multi-step denoising processing is performed on the randomly generated Gaussian noise data, and then the action sequence can be generated. Specifically, when denoising at the (T - t)-th step, based on the input first multi-modal data, the data obtained after denoising in the previous step, and the value of (T - t), the noise to be removed at the (T - t)-th step is obtained, and the data obtained after denoising in the previous step is subtracted by the obtained noise, then the denoising at the (T - t)-th step can be completed. After all denoising steps are completed, the action sequence of the 3D digital human can be generated.

[0100] To improve the accuracy of the generated action sequence of the 3D digital human, the action sequence of the present invention can be divided into a body action sequence and a hand action sequence. In some embodiments, the first multi-modal data can be input into the limb action generation model to synchronously generate the body action sequence and the hand action sequence. However, the body actions of the 3D digital human usually affect the hand actions. If the body action sequence and the hand action sequence are generated synchronously, the body action sequence and the hand action sequence are relatively independent, which will result in poor quality of the generated hand action sequence, and the hand actions of the driven 3D digital human are quite different from those of real humans.

[0101] To improve the quality of the hand action sequence, in some embodiments, the first multi-modal data of the present invention can include a lower body walking action sequence, the limb action generation model can include a body action generation model and a hand action generation model, and the action sequence can include a body action sequence and a hand action sequence;

[0102] As Figure 5 shown, step S2 can include:

[0103] Step S21, input the first multi-modal data into the body action generation model to generate a body action sequence;

[0104] Step S22, replace the lower body walking action sequence in the first multi-modal data with the body action sequence to obtain second multi-modal data;

[0105] Step S23, input the second multi-modal data into the hand action generation model to generate a hand action sequence;

[0106] Among them, the training set of the body action generation model includes the first multi-modal data sample and the corresponding body action sequence sample, and the training set of the hand action generation model includes the second multi-modal data sample and the corresponding hand action sequence sample.

[0107] When training the body action generation model, random noise is introduced into the body action sequence sample during noise addition at the t-th step to obtain the noise-added data x t, record the random noise at the t-th step as the training data for the corresponding denoising step; when denoising at the (T - t)-th step, based on the first multi-modal data sample corresponding to the body motion sequence sample, the noisy data x t and the value of (T - t), predict the random noise introduced at the t-th step, calculate the loss between the predicted noise and the actual random noise introduced at the t-th step, adjust the parameters of the body motion generation model based on the loss calculated for each denoising step, and perform denoising and loss calculation again. Repeat the above steps multiple times until the body motion generation model converges in denoising.

[0108] When training the hand motion generation model, introduce random noise to the hand motion sequence sample at the t-th step of adding noise to obtain the noisy data x t , record the random noise at the t-th step as the training data for the corresponding denoising step; when denoising at the (T - t)-th step, based on the second multi-modal data sample corresponding to the hand motion sequence sample, the noisy data x t and the value of (T - t), predict the random noise introduced at the t-th step, calculate the loss between the predicted noise and the actual random noise introduced at the t-th step, adjust the parameters of the hand motion generation model based on the loss calculated for each denoising step, and perform denoising and loss calculation again. Repeat the above steps multiple times until the body motion generation model converges in denoising.

[0109] Exemplarily, the first multi-modal data may include speech data, key motion data, style type data, and lower body walking motion sequence, and the second multi-modal data includes speech data, key motion data, style type data, and body motion sequence.

[0110] The generated hand motion sequence is matched with the body motion sequence, improving the quality of the hand motion sequence. After driving the 3D digital human based on the motion sequence, the 3D digital human can make coordinated body motions, walking motions, and hand motions.

[0111] Of course, both the body motion generation model and the hand motion generation model can be diffusion models. In this case:

[0112] As Figure 6 shown, step S21 may include:

[0113] Generate Gaussian noise data;

[0114] Use the first multi-modal data as the condition for the body motion generation model to perform denoising on the Gaussian noise data to generate the body motion sequence of the 3D digital human;

[0115] As Figure 7 shown, step S23 may include:

[0116] Using the second multi-modal data as a condition for the hand gesture generation model, denoise the Gaussian noise data to generate the hand gesture sequence of the 3D digital human.

[0117] In some embodiments, the diffusion model of the present invention may be a UNet network architecture. The UNet network consists of several encoding blocks and decoding blocks, and each encoding block or decoding block includes a Convolutional Neural Networks (CNN) and a Transformer neural network. That is to say, both the body gesture generation model and the hand gesture generation model may be a UNet network architecture based on a convolutional neural network and a Transformer neural network. As Figure 8 shown, the UNet network architecture is a left-right symmetric U-shaped structure, which may include 4 layers of structures and 8 sub-networks, and each sub-network consists of a CNN module and a Transformer module.

[0118] In the present invention, considering that the speech data exists in the form of audio and cannot be directly input into the convolutional neural network and the Transformer neural network, the speech data needs to be transformed. For this reason, in step S2 of the present invention, inputting the first multi-modal data into the trained body gesture generation model may include:

[0119] Using a speech recognition model to extract the speech features of the speech data;

[0120] Inputting the speech features into the body gesture generation model.

[0121] In some embodiments, the speech recognition model may be a Hubert (Hidden-unit Bidirectional Encoder Representations from Transformers) model or a Wav2Vec model. Among them, the process of the Hubert model extracting speech features includes: inputting the speech data is first converted into speech features through a feature extractor (such as MFCC or a convolutional neural network); using a clustering algorithm (such as k-means) to cluster the speech features to generate pseudo-labels, and these pseudo-labels represent the hidden units in the speech (such as phonemes or sub-word units); randomly masking the input features, and then training the model to predict the pseudo-labels of the masked part. Hubert uses a Transformer encoder to model the context information of the speech data, and can capture the long-range dependencies in the speech through a multi-layer self-attention mechanism. By predicting the pseudo-labels of the masked part, Hubert can learn useful representations of the speech without relying on manually labeled data.

[0122] In some embodiments, before step S2, the method for generating a 3D digital human action sequence of the present invention further includes: establishing an initial limb action generation model, a training set and a test set for the limb action generation model; training the initial limb action generation model using the training set; and testing the trained limb action generation model using the test set.

[0123] Before step S1, the method for generating a 3D digital human action sequence provided by the present invention further includes the following steps: receiving a line text; and converting the line text into speech data using speech synthesis technology.

[0124] Based on the foregoing description, it can be seen that the first multi-modal data of the 3D digital human received by the method for generating a 3D digital human action sequence provided by the present invention includes speech data. In some embodiments, the speech data can be directly uploaded by the user as an audio file, or can be collected and recorded through an audio capture device, or can also be generated through speech synthesis technology.

[0125] A common speech synthesis technology is TTS (Text To Speech) technology, which can convert the line text (characters) into speech data (natural speech stream) through an artificial neural network, realizing the real-time conversion of the line text.

[0126] The present invention allows the user to input the line text, and after automatically converting the line text into speech data using speech synthesis technology, as one type of data in the first multi-modal data, it is input into the trained limb action generation model together with other data in the first multi-modal data, thereby generating a 3D digital human action sequence.

[0127] As Figure 9 shown, the 3D digital human driving method provided by the present invention includes:

[0128] Step S3, applying the foregoing method for generating a 3D digital human action sequence to generate a 3D digital human action sequence;

[0129] Step S4, performing redirection processing on the 3D digital human action sequence;

[0130] Step S5, driving the 3D digital human using the 3D digital human action sequence after redirection processing.

[0131] The present invention uses multi-modal data including speech data to train the limb action generation model, and the obtained limb action generation model has higher accuracy, the generated action sequence has better quality and smaller gap from the real human action sequence. After using the action sequence to drive the 3D digital human, the actions of the 3D digital human can be closer to real humans.

[0132] As Figure 10As shown, the 3D digital human video generation method provided by the present invention includes:

[0133] Step S6: Driving the 3D digital human by applying the aforementioned 3D digital human driving method;

[0134] Step S7: Rendering the 3D digital human to generate a 3D digital human video.

[0135] The present invention uses multi-modal data including voice data to train a limb movement generation model. The obtained limb movement generation model has higher accuracy, the generated action sequence has better quality and a smaller gap from the real human action sequence. Using the action sequence to drive the 3D digital human and generate a 3D digital human video can make the 3D digital human video more realistic.

[0136] As Figure 11 shown, the 3D digital human action sequence generation device of the present invention includes:

[0137] A receiving module, configured to receive first multi-modal data of the 3D digital human, where the first multi-modal data includes voice data;

[0138] A generating module, configured to input the first multi-modal data into a trained limb movement generation model to generate an action sequence of the 3D digital human;

[0139] Wherein, the training set of the limb movement generation model includes first multi-modal data samples and corresponding action sequence samples.

[0140] It should be noted that the 3D digital human action sequence generation device provided by the present invention, when specifically operating, can execute the 3D digital human action sequence generation method described in any of the above embodiments, and this embodiment will not be elaborated here.

[0141] Figure 12 is a schematic structural diagram of an electronic device provided by the present invention. The electronic device may include: a processor, a communication interface, a memory, and a communication bus. Among them, the processor, the communication interface, and the memory complete mutual communication through the communication bus. The processor can call logic instructions in the memory to execute the 3D digital human action sequence generation method, the 3D digital human driving method, or the 3D digital human video generation method.

[0142] In addition, when the logical instructions in the above-mentioned memory are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs that can store program codes.

[0143] On the other hand, the present invention also provides a computer program product. The computer program product includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the 3D digital human motion sequence generation method, the 3D digital human driving method, or the 3D digital human video generation method provided in the above-mentioned various embodiments.

[0144] On another aspect, the present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it is implemented to execute the 3D digital human motion sequence generation method, the 3D digital human driving method, or the 3D digital human video generation method provided in the above-mentioned various embodiments.

[0145] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. One can select some or all of the modules according to actual needs to achieve the purpose of the solution of this embodiment. A person of ordinary skill in the art can understand and implement it without creative labor.

[0146] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on such an understanding, the essence of the above technical solution, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0147] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for generating a 3D digital human action sequence, characterized in that, Including: Receiving first multimodal data of a 3D digital human, where the first multimodal data includes speech data; Inputting the first multimodal data into a trained limb movement generation model to generate a movement sequence of the 3D digital human; Wherein, the training set of the limb movement generation model includes first multimodal data samples and corresponding movement sequence samples.

2. The method for generating a 3D digital human motion sequence according to claim 1, wherein The first multimodal data further includes at least one of key movement data, style type data, and lower body walking movement sequences.

3. The method for generating a 3D digital human action sequence according to claim 1, characterized in that, The limb movement generation model is a diffusion model; The step of inputting the first multimodal data into a trained limb movement generation model to generate a movement sequence of the 3D digital human includes: Generating Gaussian noise data; Using the first multimodal data as a condition for the diffusion model to denoise the Gaussian noise data to generate a movement sequence of the 3D digital human.

4. The method for generating a 3D digital human motion sequence according to claim 1, characterized in that, The first multimodal data includes a lower body walking movement sequence, the limb movement generation model includes a body movement generation model and a hand movement generation model, and the movement sequence includes a body movement sequence and a hand movement sequence; The step of inputting the first multimodal data into a trained limb movement generation model to generate a movement sequence of the 3D digital human includes: Inputting the first multimodal data into the body movement generation model to generate the body movement sequence; Replacing the lower body walking movement sequence in the first multimodal data with the body movement sequence to obtain second multimodal data; Inputting the second multimodal data into the hand movement generation model to generate the hand movement sequence; Wherein, the training set of the body movement generation model includes the first multimodal data samples and corresponding body movement sequence samples, and the training set of the hand movement generation model includes second multimodal data samples and corresponding hand movement sequence samples.

5. The method for generating a 3D digital human motion sequence according to claim 3, wherein The diffusion model is a UNet network architecture, and the UNet network consists of several encoding blocks and decoding blocks, and each encoding block or decoding block contains a convolutional neural network and a Transformer neural network.

6. The method for generating a 3D digital human motion sequence according to claim 1, wherein The step of inputting the first multimodal data into a trained limb movement generation model includes: Using a speech recognition model to extract speech features of the speech data; Inputting the speech features into the limb movement generation model.

7. The method for generating a 3D digital human motion sequence according to claim 1, wherein Before receiving the first multimodal data of the 3D digital human, the method further includes: Receiving a line text; Using speech synthesis technology to convert the line text into the speech data.

8. A 3D digital human driving method, characterized in that, Including: Applying the method for generating a 3D digital human movement sequence according to any one of claims 1 - 7 to generate a 3D digital human movement sequence; Performing redirection processing on the 3D digital human movement sequence; Using the 3D digital human movement sequence after redirection processing to drive the 3D digital human.

9. A method for generating 3D digital human videos, characterized in that, Including: Applying the 3D digital human driving method according to claim 8 to drive the 3D digital human; Rendering the 3D digital human to generate a 3D digital human video.

10. A generating device for 3D digital human action sequences, characterized in that, Including: A receiving module, configured to receive first multimodal data of a 3D digital human, where the first multimodal data includes speech data; A generation module, configured to input the first multi-modal data into a trained limb movement generation model to generate an action sequence of the 3D digital human; Wherein, the training set of the limb movement generation model includes first multi-modal data samples and corresponding action sequence samples.

11. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein, When the processor executes the computer program, it implements the method for generating the 3D digital human action sequence according to any one of claims 1 to 7, or the 3D digital human driving method according to claim 8, or the 3D digital human video generation method according to claim 9.

12. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method for generating the 3D digital human action sequence according to any one of claims 1 to 7, or the 3D digital human driving method according to claim 8, or the 3D digital human video generation method according to claim 9.

13. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it is the method for generating the 3D digital human action sequence according to any one of claims 1 to 7, or the 3D digital human driving method according to claim 8, or the 3D digital human video generation method according to claim 9.

Images