Voice-driven 3D digital human motion generation method, system, device and medium

Abstract

The present invention discloses a voice-driven 3D digital human motion generation method, system, device and medium, the method includes: obtaining a target input signal, the target input signal is audio, or audio and motion sequence; inputting the target input signal into a motion generation model, generating a 3D digital human motion sequence corresponding to the target input signal; the motion generation model includes an audio encoder, a motion style encoder and a motion generator, the audio encoder and the motion style encoder are processed in parallel, and a first feature vector sequence corresponding to the audio and a second feature vector corresponding to the motion sequence are generated respectively, when no motion sequence is input, the second feature vector is a zero feature vector, and the motion generator generates a 3D digital human motion sequence based on the first feature vector sequence and the second feature vector. The present invention has the characteristics of real-time performance, reduced labor costs, and free control of the generated motion sequence style.

Description

Technical Field

[0001] The present invention relates to the technical field of 3D digital human, and more specifically to a method, system, device and medium for generating 3D digital human motions driven by voice. Background Art

[0002] 3D digital human is a hyper-realistic human simulation. With the rise of the metaverse, how to drive 3D digital human to interact with users has become a problem that many people are concerned about. In order to achieve a better interactive experience, the driving technology is often required to be real-time and diversified.

[0003] Although 3D digital humans have appeared in the public eye, existing technologies usually use animations pre-made by animators to drive digital humans, resulting in high manpower and time costs. The rhythm of the animations made in this way often cannot match the language spoken by the digital humans, and there is also a single action style.

[0004] Meanwhile, in the prior art, a method for driving digital human body expressions through voice (CN116880695A) mainly drives facial body expressions, rather than body movements. A control system and method for a multi-purpose digital human (CN117076755A), a method, device, electronic device and storage medium for interactive control of a digital human (CN116841391A), and a method and device for driving a digital human (CN116485960A) can generate lip movement information and body movement information according to the interactive content, but cannot control the style of the generated movement according to a sequence of movements. A method and system for driving a sign language digital human based on a large language model (CN116959119A) selects from a pre-set movement sequence and smoothes it through an interpolation method, which is not a generative task. A method and related device for jointly generating facial and body movement parameters (CN116977499A) generates facial movements and body movements related to facial movements through audio and preset style data. This patent is based on a diffusion model and is generally not real-time. The speech-driven digital human generation method based on neural field (CN116825127A) directly generates 2D digital human video instead of driving 3D digital human.

[0005] Therefore, how to provide a voice-driven 3D digital human motion generation method, system, device and medium to overcome the above-mentioned drawbacks is a problem that technical personnel in this field urgently need to solve. Summary of the invention

[0006] In view of this, the present invention provides a voice-driven 3D digital human motion generation method, system, device and medium, which are real-time, reduce labor costs, and can freely control the style of the generated motion sequence.

[0007] In order to achieve the above object, the present invention adopts the following technical solution:

[0008] In a first aspect, the present invention provides a method for generating 3D digital human motions driven by speech, comprising:

[0009] Acquire a target input signal, where the target input signal is audio, or audio and action sequence;

[0010] Input the target input signal into the action generation model to generate a 3D digital human action sequence corresponding to the target input signal;

[0011] The action generation model includes an audio encoder, an action style encoder and an action generator. The audio encoder and the action style encoder are processed in parallel to generate a first feature vector sequence corresponding to the audio and a second feature vector corresponding to the action sequence respectively. When there is no action sequence input, the second feature vector is a zero feature vector. The action generator generates a 3D digital human action sequence based on the first feature vector sequence and the second feature vector.

[0012] Preferably, the audio encoder comprises in sequence: a feature extraction module, two one-dimensional convolutional layers and a fully connected layer, each one-dimensional convolutional layer being followed by a nonlinear activation function;

[0013] The audio passes through the feature extraction module to obtain the original feature vector of each frame;

[0014] The original feature vector passes through two one-dimensional convolutional layers in sequence. After each one-dimensional convolutional layer, it passes through a nonlinear activation function, and finally passes through a fully connected layer to obtain the final output audio feature vector sequence.

[0015] Preferably, the action style encoder comprises, in sequence: two one-dimensional convolutional layers, a position encoding module and a feedforward module;

[0016] The action sequence passes through two one-dimensional convolutional layers in sequence, where each convolutional layer is followed by a nonlinear activation function and layer normalization operation to obtain the initial feature vector;

[0017] A position encoding module is introduced into the initial feature vector, and the output after position encoding is input into the feedforward module;

[0018] The multi-dimensional Gaussian distribution is obtained through the feedforward module. The specific processing formula is:

[0019]

[0020] Among them, the mean μ and variance σ 2 They are all multidimensional vectors, with the same dimension as the feature vector output by the audio encoder, N(x|μ,σ 2) is the mathematical expression of Gaussian distribution, which means that the mean is μ and the variance is σ 2 Gaussian distribution, x represents a random vector that conforms to Gaussian distribution.

[0021] Preferably, the action generator comprises, in sequence: three layers of fully connected layers and two layers of gated recurrent units;

[0022] The second eigenvector is processed by three fully connected layers in sequence to obtain the hidden state variable;

[0023] Each frame in the first feature vector sequence is merged with the second feature vector and then used together with the latent state variable as the time sequence input of the two-layer gated recurrent unit to generate an action sequence corresponding to the number of audio frames.

[0024] Preferably, the action generation model loss function L is:

[0025]

[0026] Among them, m represents the number of training samples in a batch during training, and y i represents the i-th action sequence predicted based on the i-th audio in a training batch, represents the real action sequence corresponding to the i-th audio, D KL (·) represents the KL divergence between the multi-dimensional Gaussian distribution predicted by the action style encoder and the standard Gaussian distribution.

[0027] In a second aspect, the present invention provides a voice-driven 3D digital human motion generation system, comprising:

[0028] Data acquisition module: used to acquire target input signal, which is audio, or audio and action sequence;

[0029] Action generation module: used to input the target input signal into the action generation model to generate a 3D digital human action sequence corresponding to the target input signal;

[0030] The action generation model includes an audio encoder, an action style encoder and an action generator. The audio encoder and the action style encoder are processed in parallel to generate a first feature vector sequence corresponding to the audio and a second feature vector corresponding to the action sequence respectively. When there is no action sequence input, the second feature vector is a zero feature vector. The action generator generates a 3D digital human action sequence based on the first feature vector sequence and the second feature vector.

[0031] Preferably, the audio encoder comprises in sequence: a feature extraction module, two one-dimensional convolutional layers and a fully connected layer, each one-dimensional convolutional layer being followed by a nonlinear activation function;

[0032] The feature extraction module is used to extract features from the audio and obtain the original feature vector of each frame;

[0033] The original feature vector passes through two one-dimensional convolutional layers in sequence. After each one-dimensional convolutional layer, it passes through a nonlinear activation function, and finally passes through a fully connected layer to obtain the final output audio feature vector sequence.

[0034] Preferably, the action style encoder comprises, in sequence: two one-dimensional convolutional layers, a position encoding module and a feedforward module;

[0035] The action sequence passes through two one-dimensional convolutional layers in sequence, where each convolutional layer is followed by a nonlinear activation function and layer normalization operation to obtain the initial feature vector;

[0036] A position encoding module is introduced into the initial feature vector, and the output after position encoding is input into the feedforward module;

[0037] The feedforward module is used to calculate the multi-dimensional Gaussian distribution. The specific processing formula is:

[0038]

[0039] Among them, the mean μ and variance σ 2 They are all multidimensional vectors, with the same dimension as the feature vector output by the audio encoder, N(x|μ,σ 2 ) is the mathematical expression of Gaussian distribution, which means that the mean is μ and the variance is σ 2 Gaussian distribution, x represents a random vector that conforms to Gaussian distribution.

[0040] Preferably, the action generator comprises, in sequence: three layers of fully connected layers and two layers of gated recurrent units;

[0041] The second eigenvector is processed by three fully connected layers in sequence to obtain the hidden state variable;

[0042] Each frame in the first feature vector sequence is merged with the second feature vector and then used together with the latent state variable as the time sequence input of the two-layer gated recurrent unit, and finally an action sequence corresponding to the number of audio frames is generated.

[0043] Preferably, the action generation model loss function L is:

[0044]

[0045] Among them, m represents the number of training samples in a batch during training, and y i represents the i-th action sequence predicted based on the i-th audio in a training batch, represents the real action sequence corresponding to the i-th audio, D KL(·) represents the KL divergence between the multi-dimensional Gaussian distribution predicted by the action style encoder and the standard Gaussian distribution.

[0046] In a third aspect, the present invention provides a computer device, comprising:

[0047] Memory for storing computer programs;

[0048] A processor is used to implement the steps of the voice-driven 3D digital human motion generation method when executing the computer program.

[0049] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the voice-driven 3D digital human motion generation method are implemented.

[0050] The present invention has the following effects:

[0051] 1) This method is real-time. Many current methods related to motion generation require a long inference time and cannot meet the needs of real-time interaction between 3D digital humans and users. However, all network structures used in this method are lightweight network structures, and the audio encoder and motion style encoder work in parallel, which makes the inference speed of this method fast and can meet the needs of real-time interaction.

[0052] 2) This method can reduce labor costs. After using this method, no additional labor is needed to produce 3D digital human animation, which can be directly generated according to audio. When there is an action sequence input, the action sequence can be reused, which saves a lot of manpower accordingly.

[0053] 3) This method can optionally input any pre-made action sequence. Most previous methods cannot control the style of the generated action, or require explicit input of the required style. Such methods have low diversity and generalization. The action style encoder of this method implicitly extracts the style of the input action sequence and incorporates it into the generated action sequence, allowing users to freely control the style of the generated action sequence, making the generated actions richer and more vivid, and improving diversity and generalization.

[0054] 4) This method can generate action sequences that match the rhythm of the audio. When facing different audio inputs, it is difficult to align the rhythm and rhythm of the action created by the action artist with the audio, resulting in unnatural movements of the 3D digital human. If alignment is required, the action needs to be modified for each audio, which is time-consuming and labor-intensive. The actions generated by this method can automatically align with the rhythm and rhythm of the audio, making the generated actions more natural. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0056] Figure 1 A flow chart of a voice-driven 3D digital human motion generation method provided by the present invention.

[0057] Figure 2 This is a schematic diagram of the structure of the audio encoder provided by the present invention.

[0058] Figure 3 This is a schematic diagram of the structure of the action style encoder provided by the present invention.

[0059] Figure 4 This is a schematic diagram of the feedforward module structure provided by the present invention.

[0060] Figure 5 This is a schematic diagram of the structure of the action generator provided by the present invention.

[0061] Figure 6 This is a structural block diagram of a voice-driven 3D digital human motion generation system provided by the present invention. DETAILED DESCRIPTION

[0062] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0063] The embodiment of the present invention discloses a method for generating 3D digital human motion driven by voice, such as Figure 1 As shown, including:

[0064] Get a target input signal, which is audio, or audio and action sequence. The audio can be a real human voice, that is, recorded audio, or generated audio, such as using a common text-to-speech (TTS) algorithm to generate audio. The action sequence can be any length of action, as long as the duration of the action sequence is not less than 1 second.

[0065] Input the target input signal into the action generation model to generate a 3D digital human action sequence corresponding to the target input signal;

[0066] The action generation model includes an audio encoder, an action style encoder and an action generator. The audio encoder generates a first feature vector sequence corresponding to the audio, and the action style encoder generates a second feature vector corresponding to the action sequence. When there is no action sequence input, the second feature vector is a zero feature vector. The action generator generates a 3D digital human action sequence with the same duration as the audio based on the first feature vector sequence and the second feature vector. Among them, the audio encoder and the action style encoder are parallel on the input side, and will be input into the action generator together on the output side. It should be emphasized here that the action sequence input is optional. When the action sequence is not input, the action style encoder will still output the second feature vector. At this time, the second feature vector is a feature vector with all zeros. Among them, the prefabricated action sequence and the 3D digital human action sequence both represent: a sequence of rotation values ​​(usually expressed in the form of quaternions) of each skeletal node of the 3D digital human.

[0067] When used, this method takes a segment of audio as input, and optionally inputs a pre-made action sequence as a condition to control the style of the generated action. The final output is a 3D digital human action sequence. If a pre-made action sequence is input, the style of the output 3D digital human action sequence will be the same as the input pre-made action sequence. The action style mentioned here can be a manifestation of emotion, such as high-frequency hand movements when excited, and low-frequency body movements when frustrated. It can also be a manifestation of age, such as the movements of the elderly can always be hunched. In general, the typical features of a segment of action can be regarded as the style of the action. This method can incorporate the abstract style information in the input action sequence into the generated 3D digital human action sequence.

[0068] This method can be generally divided into three modules, namely audio encoder, action style encoder, and action generator. The following will introduce these three modules in turn.

[0069] The audio encoder is used to convert the audio into a first feature vector sequence suitable for action generation. An implementation structure of the audio encoder is as follows: Figure 2 As shown in the figure. The original audio needs to be feature extracted before it can be input into the neural network in the audio encoder. Specifically, the logarithmic amplitude in the Mel spectrum of the audio and the corresponding energy of each frame are extracted here as the audio features and input into the subsequent network. The network here consists of two one-dimensional convolutional layers and one fully connected layer, where each convolutional layer is followed by a nonlinear activation function. Finally, the audio encoder will output a multi-frame, fixed-length audio feature vector sequence for subsequent modules.

[0070] The action style encoder is used to encode the prefabricated action sequence into a second feature vector of fixed length, which will contain a low-dimensional representation of the style of the action sequence. An implementation structure of the action style encoder is as follows Figure 3As shown in Figure 1. The action sequence will first pass through two layers of one-dimensional convolutional layers, where each convolutional layer is followed by a nonlinear activation function and layer normalization operation. Then position encoding is introduced to allow the network to have cognition of the sequence order. The output after position encoding will be input into the feedforward module, which is similar to the module in the common network structure Transformer. An implementation structure is as follows Figure 4 The function of the feedforward module is to predict the mean and variance of a multidimensional Gaussian distribution, which can be expressed mathematically as follows:

[0071]

[0072] Among them, the mean μ and variance σ 2 are all multidimensional vectors, with the same dimension as the feature vector output by the audio encoder. So with the mean and variance, we can get a multidimensional Gaussian distribution, so that we can randomly sample a feature vector as the output of the action style encoder based on this distribution, and the second feature vector has the same length as the first feature vector output by the audio encoder. Since the use of the action style encoder is optional, if there is no pre-made action sequence as input, the output of the action style encoder will be a feature vector of all zeros.

[0073] The action generator is used to generate a final 3D digital human action sequence according to the first feature vector sequence output by the audio encoder and the second feature vector generated by the action style encoder. An implementation structure is as follows: Figure 5 As shown. Here, the second feature vector output by the action style encoder is called the action feature vector, and there is only one, while the first feature vector sequence output by the audio encoder is called the audio feature vector sequence. The sequence contains n frames, that is, n audio feature vectors, and the size of n is related to the sampling rate of the audio. The action feature vector will first pass through three fully connected layers, each of which is followed by a nonlinear activation function, and then generate a hidden state variable. This variable contains the initialization state information required by the gated recurrent unit (Gated Recurrent Units, GRU). At the same time, the gated recurrent unit also requires a time series input, which is actually the audio feature vector sequence. In order to include the action style information, each frame of the audio feature vector sequence is merged with the same action feature vector as the time series input of the gated recurrent unit, and finally a 3D digital human action sequence corresponding to the number of audio frames is generated.

[0074] The overall structure of this method is lightweight, so the network inference speed is fast and can achieve real-time effects on common GPUs. Before using this method, you need to use training data to train and optimize the above-mentioned overall network. Each training sample used in this method consists of a piece of audio and its corresponding target action sequence. The number of frames of the audio is required to be exactly the same as the number of frames of the target action sequence, and the action is consistent with the rhythm and rhythm of the audio. The action sequence needs to be additionally labeled with its style. If you choose to use an action style encoder during training, you should choose an action sequence with the same style as the target action sequence as input. The number of frames of the action sequence used as input is random. It is worth noting that after training is completed, the action sequence used as input can be any action sequence, not necessarily the action sequence in the training data set. The loss function used to optimize the overall network parameters during training is as follows:

[0075]

[0076] Where m means there are m training samples in a batch during training, y i represents the i-th action sequence predicted based on the i-th audio in a training batch, represents the real action sequence corresponding to the i-th audio. KL (·) indicates the calculation of the KL divergence (Kullback-Leibler divergence), which is the KL divergence between the multi-dimensional Gaussian distribution predicted by the action style encoder and the standard Gaussian distribution.

[0077] The embodiment of the present invention provides a voice-driven 3D digital human motion generation system, such as Figure 6 As shown, including:

[0078] Data acquisition module: obtains the target input signal, which is audio, or audio and action sequence; the audio can be real human voice, that is, recorded audio, or generated audio, for example, audio is generated using a common text-to-speech algorithm (Text-To-Speech, TTS); the action sequence can be of any length, as long as the duration of the action sequence is not less than 1 second.

[0079] Action generation module: inputs the target input signal into the action generation model to generate a 3D digital human action sequence corresponding to the target input signal;

[0080] The action generation model includes an audio encoder, an action style encoder and an action generator. The audio encoder generates a first feature vector sequence corresponding to the audio, and the action style encoder generates a second feature vector corresponding to the action sequence. When there is no action sequence input, the second feature vector is a zero feature vector. The action generator generates a 3D digital human action sequence with the same duration as the audio based on the first feature vector sequence and the second feature vector. Among them, the audio encoder and the action style encoder are parallel on the input side, and will be input into the action generator together on the output side. It should be emphasized here that the action sequence input is optional. When the action sequence is not input, the action style encoder will still output the second feature vector. At this time, the second feature vector is a feature vector with all zeros. Among them, the prefabricated action sequence and the 3D digital human action sequence both represent: a sequence of rotation values ​​(usually expressed in the form of quaternions) of each skeletal node of the 3D digital human.

[0081] When in use, this system takes a segment of audio as input, and can optionally input a pre-made action sequence as a condition to control the style of the generated action. The final output is a 3D digital human action sequence. If a pre-made action sequence is input, the style of the output 3D digital human action sequence will be the same as the input pre-made action sequence. The action style mentioned here can be a manifestation of emotion, such as high-frequency hand movements when excited, and low-frequency body movements when frustrated. It can also be a manifestation of age, such as the movements of the elderly can always be hunched. In general, the typical features of a segment of action can be regarded as the style of the action. This method can incorporate the abstract style information in the input action sequence into the generated 3D digital human action sequence.

[0082] The audio encoder is used to convert the audio into a first feature vector sequence suitable for action generation. An implementation structure of the audio encoder is as follows: Figure 2 As shown. The original audio needs to be feature extracted before it can be input into the neural network in the audio encoder. Specifically, the feature extraction module is used to extract features from the audio, obtain the logarithmic amplitude in the Mel spectrum of the audio and the corresponding energy of each frame, extract the logarithmic amplitude in the Mel spectrum of the audio and the corresponding energy of each frame as the audio features and input them into the subsequent network. The network here consists of two one-dimensional convolutional layers and one fully connected layer, where each convolutional layer is followed by a nonlinear activation function. Finally, the audio encoder will output a multi-frame, fixed-length audio feature vector sequence for subsequent modules.

[0083] The action style encoder is used to encode the prefabricated action sequence into a second feature vector of fixed length, which will contain a low-dimensional representation of the style of the action sequence. An implementation structure of the action style encoder is as follows Figure 3As shown in Figure 1. The action sequence will first pass through two layers of one-dimensional convolutional layers, where each convolutional layer is followed by a nonlinear activation function and layer normalization operation. Then position encoding is introduced to allow the network to have cognition of the sequence order. The output after position encoding will be input into the feedforward module, which is similar to the module in the common network structure Transformer. An implementation structure is as follows Figure 4 The function of the feedforward module is to predict the mean and variance of a multidimensional Gaussian distribution, which can be expressed mathematically as follows:

[0084]

[0085] Among them, the mean μ and variance σ 2 are all multidimensional vectors, with the same dimension as the feature vector output by the audio encoder. So with the mean and variance, we can get a multidimensional Gaussian distribution, so that we can randomly sample a feature vector as the output of the action style encoder based on this distribution, and the second feature vector has the same length as the first feature vector output by the audio encoder. Since the use of the action style encoder is optional, if there is no pre-made action sequence as input, the output of the action style encoder will be a feature vector of all zeros.

[0086] The action generator is used to generate a final 3D digital human action sequence according to the first feature vector sequence output by the audio encoder and the second feature vector generated by the action style encoder. An implementation structure is as follows: Figure 5 As shown. Here, the second feature vector output by the action style encoder is called the action feature vector, and there is only one, while the first feature vector sequence output by the audio encoder is called the audio feature vector sequence. The sequence contains n frames, that is, n audio feature vectors, and the size of n is related to the sampling rate of the audio. The action feature vector will first pass through three fully connected layers, each of which is followed by a nonlinear activation function, and then generate a hidden state variable. This variable contains the initialization state information required by the gated recurrent unit (Gated Recurrent Units, GRU). At the same time, the gated recurrent unit also requires a time series input, which is actually the audio feature vector sequence. In order to include the action style information, each frame of the audio feature vector sequence is merged with the same action feature vector as the time series input of the gated recurrent unit, and finally a 3D digital human action sequence corresponding to the number of audio frames is generated.

[0087] The overall structure of this system is lightweight, so the network inference speed is fast and can achieve real-time effects on common GPUs. Before using this system, you need to use training data to train and optimize the above-mentioned overall network. Each training sample used is composed of a piece of audio and its corresponding target action sequence. The number of frames of the audio is required to be exactly the same as the number of frames of the target action sequence, and the action is consistent with the rhythm and rhythm of the audio. The action sequence needs to be additionally labeled with its style. If you choose to use an action style encoder during training, you should choose an action sequence with the same style as the target action sequence as input. The number of frames of the action sequence used as input is random. It is worth noting that after training is completed, the action sequence used as input can be any action sequence, not necessarily the action sequence in the training data set. The loss function used to optimize the overall network parameters during training is as follows:

[0088]

[0089] Where m means there are m training samples in a batch during training, y i represents the i-th action sequence predicted based on the i-th audio in a training batch, represents the real action sequence corresponding to the i-th audio. KL (·) indicates the calculation of the KL divergence (Kullback-Leibler divergence), which is the KL divergence between the multi-dimensional Gaussian distribution predicted by the action style encoder and the standard Gaussian distribution.

[0090] The present invention provides a computer device, comprising:

[0091] Memory for storing computer programs;

[0092] The processor is used to implement the steps of the above-mentioned voice-driven 3D digital human motion generation method when executing a computer program.

[0093] For an introduction to a computer device provided by the present invention, please refer to the above method embodiment, and the present invention will not be described in detail here.

[0094] The computer readable storage medium in the present invention stores a computer program, and when the computer program is executed by the processor, the steps of the above-mentioned voice-driven 3D digital human motion generation method are implemented. The storage medium includes: a mobile storage device, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0095] For an introduction to the computer-readable storage medium provided by the present invention, please refer to the above method embodiment, and the present invention will not be repeated here.

[0096] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0097] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A voice-driven 3D digital human motion generation method, characterized in that: include: Obtaining a target input signal, where the target input signal is an audio and action sequence; Input the target input signal into the action generation model to generate a 3D digital human action sequence corresponding to the target input signal; The action generation model includes an audio encoder, an action style encoder and an action generator. The audio encoder and the action style encoder are processed in parallel to generate a first feature vector sequence corresponding to the audio and a second feature vector corresponding to the action sequence respectively. When there is no action sequence input, the second feature vector is a zero feature vector. The action generator generates a 3D digital human action sequence based on the first feature vector sequence and the second feature vector. The action style encoder consists of two one-dimensional convolutional layers, a position encoding module, and a feedforward module. The action sequence passes through two one-dimensional convolutional layers in sequence, where each convolutional layer is followed by a nonlinear activation function and layer normalization operation to obtain the initial feature vector; The position encoding module is introduced into the initial feature vector, and the output after position encoding is input into the feedforward module; the feedforward module is used to calculate the multi-dimensional Gaussian distribution, and the specific processing formula is: Among them, the mean μ and variance σ 2 They are all multidimensional vectors, with the same dimension as the feature vector output by the audio encoder, N(x|μ,σ 2 ) is the mathematical expression of Gaussian distribution, which means that the mean is μ and the variance is σ 2 Gaussian distribution, x represents a random vector that conforms to Gaussian distribution; The action generator consists of three fully connected layers and two gated recurrent units in sequence; The second eigenvector is processed by three fully connected layers in sequence to obtain the hidden state variable; Each frame in the first feature vector sequence is merged with the second feature vector and then used together with the latent state variable as the time sequence input of the two-layer gated recurrent unit to generate an action sequence corresponding to the number of audio frames.

2. The method for generating 3D digital human motion driven by voice according to claim 1, characterized in that: The audio encoder includes: a feature extraction module, two one-dimensional convolutional layers and a fully connected layer, and each one-dimensional convolutional layer is followed by a nonlinear activation function; The audio passes through the feature extraction module to obtain the original feature vector of each frame; The original feature vector passes through two one-dimensional convolutional layers in sequence. After each one-dimensional convolutional layer, it passes through a nonlinear activation function, and finally passes through a fully connected layer to obtain the final output audio feature vector sequence.

3. The method for generating 3D digital human motion driven by voice according to claim 1, characterized in that: The loss function L of the action generation model is: Among them, m represents the number of training samples in a batch during training, and y i represents the i-th action sequence predicted based on the i-th audio in a training batch, represents the real action sequence corresponding to the i-th audio, D KL (·) represents the KL divergence between the multi-dimensional Gaussian distribution predicted by the action style encoder and the standard Gaussian distribution.

4. A voice-driven 3D digital human motion generation system, characterized in that: include: Data acquisition module: used to acquire target input signals, which are audio and action sequences; Action generation module: used to input the target input signal into the action generation model to generate a 3D digital human action sequence corresponding to the target input signal; The action generation model includes an audio encoder, an action style encoder and an action generator. The audio encoder and the action style encoder are processed in parallel to generate a first feature vector sequence corresponding to the audio and a second feature vector corresponding to the action sequence respectively. When there is no action sequence input, the second feature vector is a zero feature vector. The action generator generates a 3D digital human action sequence based on the first feature vector sequence and the second feature vector. The action style encoder consists of two one-dimensional convolutional layers, a position encoding module, and a feedforward module. The action sequence passes through two one-dimensional convolutional layers in sequence, where each convolutional layer is followed by a nonlinear activation function and layer normalization operation to obtain the initial feature vector; The position encoding module is introduced into the initial feature vector, and the output after position encoding is input into the feedforward module; the feedforward module is used to calculate the multi-dimensional Gaussian distribution, and the specific processing formula is: Among them, the mean μ and variance σ 2 They are all multidimensional vectors, with the same dimension as the feature vector output by the audio encoder, N(x|μ,σ 2 ) is the mathematical expression of Gaussian distribution, which means that the mean is μ and the variance is σ 2 Gaussian distribution, x represents a random vector that conforms to Gaussian distribution; The action generator consists of three fully connected layers and two gated recurrent units in sequence; The second eigenvector is processed by three fully connected layers in sequence to obtain the hidden state variable; Each frame in the first feature vector sequence is merged with the second feature vector and then used together with the latent state variable as the time sequence input of the two-layer gated recurrent unit to generate an action sequence corresponding to the number of audio frames.

5. The voice-driven 3D digital human motion generation system according to claim 4, characterized in that: The audio encoder includes: a feature extraction module, two one-dimensional convolutional layers and a fully connected layer, and each one-dimensional convolutional layer is followed by a nonlinear activation function; The feature extraction module is used to extract features from the audio and obtain the original feature vector of each frame; The original feature vector passes through two one-dimensional convolutional layers in sequence. After each one-dimensional convolutional layer, it passes through a nonlinear activation function, and finally passes through a fully connected layer to obtain the final output audio feature vector sequence.

6. The voice-driven 3D digital human motion generation system according to claim 4, characterized in that: The loss function L of the action generation model is: Among them, m represents the number of training samples in a batch during training, and y i represents the i-th action sequence predicted based on the i-th audio in a training batch, represents the real action sequence corresponding to the i-th audio, D KL (·) represents the KL divergence between the multi-dimensional Gaussian distribution predicted by the action style encoder and the standard Gaussian distribution.

7. A computer device, characterized in that: include: Memory for storing computer programs; A processor is used to implement the steps of a voice-driven 3D digital human motion generation method as described in any one of claims 1 to 3 when executing the computer program.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of a voice-driven 3D digital human motion generation method as claimed in any one of claims 1 to 3.

Images