A digital human driving and presentation system and method for enhanced emotion

Abstract

The present invention discloses an emotion-enhanced digital human driving and presentation system and method, belonging to the field of computer technology. The system includes an audio decoupling module, a potential diffusion model module, a human motion prior module, a training module, and a rendering image generation module. The system generates facial expressions and gesture action sequences through voice-driven, predicts and reconstructs the facial expressions and gesture action sequences to obtain the final action sequence; and generates a neural rendering image based on the final action sequence. The present invention captures the emotion and style features in the voice through voice input, introduces a cross-attention feature fusion mechanism to enhance the expression of emotion and personal style features, and expands the voice-driven gesture generation task to the facial expression and gesture action synchronization generation task, thereby achieving the integrity of the generated digital human actions and greatly improving the emotional expressiveness of the digital human facial expressions and gesture actions.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a system and method for driving and presenting a digital human with enhanced emotions. Background Art

[0002] In the intersection of artificial intelligence and computer graphics, digital human technology has been a hot topic of research in recent years. Digital human technology refers to the use of computer graphics, artificial intelligence, computer vision, speech recognition, natural language processing and other technologies to simulate human behavior, expression, voice and other characteristics to form a virtual character with human appearance and behavior characteristics. With the rapid development of technologies such as virtual reality (VR), augmented reality (AR), 3D animation, game development, film production and intelligent customer service, higher requirements are placed on the realism and interactive capabilities of digital humans.

[0003] Voice-driven digital human facial expression and gesture generation plays a core role in the field of digital humans. It enables digital humans to generate realistic facial expressions and precise gestures based on voice input, greatly enhancing the interactivity and realism of digital humans and improving user experience. The development of this technology can not only reduce the driving costs in fields such as VR, virtual live broadcasts and games, but also empower the digital transformation of industries such as education, medical care and intelligent services in terms of cross-domain integration and expansion.

[0004] For example, the invention application with publication number CN116977499A discloses a method for jointly generating facial and body motion parameters and related equipment, which obtains audio data and preset style data; generates motion parameters based on the audio data and style data, wherein the motion parameters include facial motion parameters and body motion parameters, and the body motion parameters are generated based on the facial motion parameters; and drives the target based on the motion parameters. This method can jointly generate facial motion parameters and body motion parameters based on audio data and style data, and refer to the facial motion parameters when generating the body motion parameters, so that the effects of audio-driven facial expressions and body movements are synchronized, close to the real motion state of the face and body.

[0005] For another example, the invention application with publication number CN118969009A discloses a method for synthesizing interactive digital humans based on voice-driven artificial intelligence. The method collects voice data submitted from different channels and platforms through a data acquisition unit, and pre-processes the input raw audio data, including voice filtering and voice feature digitization. Based on a deep learning model, the audio features are extracted from the input voice signal, and the facial key point motion weights and gesture and head animation parameters are predicted. Then, facial expressions, smooth gestures, and head movements that are precisely synchronized with the digital human's language broadcast content are generated, so as to realize the driving and natural interaction of the digital human, thereby realizing the synchronous mapping of the semantics and emotional expression contained in the voice to the facial gestures and body movement rendering of the virtual human 3D model, thereby enhancing the authenticity and professionalism of the digital human's emotions.

[0006] However, in existing work, there are some limitations in the speech-driven digital human facial expression and gesture generation methods. These methods often focus on independently generating speech-driven expressions or gestures, while ignoring the integrity of digital human action generation. In addition, existing studies usually do not consider the impact of emotional and stylistic features in speech on expressions and gestures, resulting in the lack of emotional expressiveness and personalization of generated facial expressions and gestures. Summary of the invention

[0007] The purpose of the present invention is to provide an emotion-enhanced digital human driving and presentation system and method, which is used to solve the problems that the existing voice-driven facial expression and gesture generation is incomplete, and the generated facial expressions and gestures lack emotional expressiveness and personalization.

[0008] To achieve the above-mentioned purpose of the invention, an embodiment provides an emotion-enhanced digital human driving and presentation system, comprising an audio decoupling module, a potential diffusion model module, a human motion prior module, a training module, and a rendering image generation module;

[0009] The audio decoupling module includes three independent encoders for decomposing the original audio sequence into three independent feature latent vectors, represented as a content latent vector, an emotion latent vector and a personal style latent vector;

[0010] The latent diffusion model module is used to convert the original action sequence into motion noise in a high-dimensional space by adding noise, and then convert the content latent vector, the emotion latent vector, the personal style latent vector, the diffusion time step encoding and the motion noise into a predicted motion latent vector through iterative denoising based on a fusion mechanism;

[0011] The human body motion prior module includes a motion prior encoder and a motion prior decoder, which are used for encoding and decoding of 3D human body motions, wherein the motion prior encoder is used to provide information about the position of each element in the motion sequence, and reconstructs the information about the position of each element in the motion sequence to generate a reconstructed motion potential vector; the motion prior decoder is used to receive the predicted motion potential vector and the reconstructed motion potential vector as input, and decode to obtain the predicted motion sequence and the reconstructed motion sequence; the motion sequence includes: a facial expression sequence or a gesture motion sequence;

[0012] The training module is used to optimize the potential diffusion model module and the human motion prior module, and align the original audio with the motion potential vector by calculating the potential diffusion model gradient and the loss function of the human motion prior module, wherein the loss function of the human motion prior module includes reconstruction loss, KL divergence loss and alignment reconstruction loss;

[0013] The rendering image generation module is used to form two-dimensional key points of plane coordinates according to the facial expression coefficients and three-dimensional human body parameters in the aligned predicted action sequence, and generate a neural rendering image with facial expression and whole body semantic information.

[0014] In one embodiment, the three independent encoders include a content encoder, a sentiment encoder, and a personal style encoder;

[0015] The content encoder is used to extract information related to speech rhythm and speech as a content latent vector;

[0016] The emotion encoder is used to extract the emotion features in the original audio sequence as the emotion latent vector;

[0017] The personal emotion encoder is used to extract features related to personal style as a personal style latent vector.

[0018] In one embodiment, the fusion-based mechanism converts the content latent vector, the emotion latent vector, the personal style latent vector, the diffusion time step coding and the motion noise into a predicted motion latent vector through iterative denoising, including: introducing a cross-attention layer in the Transformer block, concatenating the content latent vector, the emotion latent vector, the personal style latent vector, the diffusion time step coding and the motion noise according to the first dimension of the vector to obtain a new noise latent vector, concatenating the emotion latent vector and the personal style latent vector according to the first dimension of the vector to obtain a conditional latent vector, using the new noise latent vector and the conditional latent vector as inputs of the cross-attention layer, obtaining the final output through a linear layer, and iteratively denoising to obtain a predicted motion latent vector for subsequent propagation.

[0019] In one embodiment, the Transformer block includes two independent Transformer blocks, namely a facial expression Transformer block and a gesture action Transformer block, which are used to receive facial expression noise and gesture action noise for denoising.

[0020] In one embodiment, a denoiser supervision loss function is introduced in the iterative denoising process to remove noise from the motion latent vector so that the motion latent vector matches the motion noise.

[0021] In one embodiment, the motion a priori encoder and the motion a priori decoder have the same structure, both using a UNet structure with Transformer blocks as units, and there are jump connections between Transformer blocks to restore more detail information in the decoding stage.

[0022] In one embodiment, the information reconstruction encoding of each element position in the action sequence to generate a reconstructed motion latent vector includes: a motion prior encoder receives an action sequence of a facial expression sequence and a gesture action sequence as input, outputs a mean and a covariance matrix to form a normal distribution, and uses reparameterization to sample from the normal distribution to generate a reconstructed motion latent vector.

[0023] In one embodiment, the reconstruction loss includes a posture parameter reconstruction loss and a vertex coordinate reconstruction loss, wherein the posture parameter reconstruction loss is used to measure the difference between the real action sequence and the reconstructed action sequence, and the vertex coordinate reconstruction loss is used to measure the difference between the real vertex coordinates and the reconstructed vertex coordinates, wherein the action sequence is passed through the average mesh shape of the SMPLX model in three-dimensional space to obtain the vertex coordinates;

[0024] The KL divergence loss is used to measure the difference between the variational distribution and the prior distribution, so that the variational distribution is close to the prior distribution, wherein the variational distribution is a normal distribution composed of the mean and covariance matrix output by the motion prior encoder, and the prior distribution is a standard normal distribution composed of 0 and 1;

[0025] The alignment reconstruction loss includes posture parameter alignment reconstruction loss and vertex coordinate alignment reconstruction loss, wherein the posture parameter alignment reconstruction loss is used to measure the difference between the real action sequence and the predicted action sequence, and the vertex coordinate alignment reconstruction loss is used to measure the difference between the real vertex coordinates and the predicted vertex coordinates.

[0026] The present invention also provides a method for driving and presenting a digital human with enhanced emotion, comprising the following steps:

[0027] Obtain the original audio sequence, decouple the original audio sequence into content latent vector, emotion latent vector and personal style latent vector based on the audio decoupling module, combine the motion noise and diffusion time step encoding transformed by the latent diffusion model module, and iteratively denoise to obtain the predicted motion latent vector;

[0028] Encode the action sequences including the facial expression sequence and the gesture action sequence through the motion prior encoder to obtain the reconstructed motion latent vector; use the predicted motion latent vector and the reconstructed motion latent vector as the input of the motion prior decoder to decode and obtain the predicted action sequence and the reconstructed action sequence;

[0029] Based on the decoded predicted action sequence and the reconstructed action sequence, the denoiser supervision loss, reconstruction loss, KL divergence loss and alignment reconstruction loss are combined to align the motion latent vectors in the original audio sequence with the decoded predicted action sequence and the reconstructed action sequence;

[0030] The rendered image generation module is used to render the facial expression coefficients and three-dimensional human body parameters in the aligned predicted action sequence to generate a neural rendered image with facial and full-body semantic information, thus completing the driving and presentation of the digital human.

[0031] Compared with the prior art, the present invention has the following beneficial effects:

[0032] (1) By introducing a human motion prior module that includes facial expressions and gestures, the synchronous generation of facial expressions and gestures driven by voice is achieved, thus achieving the completeness of the generated digital human motions;

[0033] (2) By decoding speech into content latent vectors, emotion latent vectors and personal style latent vectors, and introducing a cross-attention feature fusion mechanism based on the latent diffusion model, the expression of emotion and style features is enhanced, so that the generated facial expressions and gestures can better integrate the corresponding emotional elements, greatly improving the emotional expressiveness of digital human facial expressions and gestures. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art are briefly introduced below.

[0035] Figure 1 A schematic diagram of the system structure for driving and presenting a digital human with enhanced emotions provided in an embodiment.

[0036] Figure 2 Schematic diagram of the diffusion denoising process.

[0037] Figure 3 Schematic diagram of the cross-attention feature fusion mechanism.

[0038] Figure 4 Schematic diagram of the human motion prior module.

[0039] Figure 5 A schematic diagram of the process of driving and presenting an emotion-enhanced digital human provided in an embodiment.

[0040] Figure 6 Schematic diagram of CCBR neural rendering image. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the following is a

[0042] The present invention is further described in detail in the following examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and do not limit the scope of protection of the present invention.

[0043] To ensure the integrity of the digital human's generated movements and emotional expression, the embodiment provides a system for driving and presenting a digital human with enhanced emotions, such as Figure 1 As shown in Figure 1, the system consists of five components: audio decoupling module, potential diffusion model module, human motion prior module, training module, and rendered image generation module. The following will introduce each part of the basic model architecture in detail.

[0044] Audio decoupling module: As a key part of the speech-driven facial expression and gesture generation task, it is responsible for converting the input raw audio sequence Decomposed into three independent latent representations: content, emotion, and personal style. This module has a dedicated encoder architecture consisting of three independent encoders. Each encoder is responsible for extracting a latent representation from the input speech signal. Among them, the content encoder is responsible for extracting information related to speech rhythm and speech, the emotion encoder is responsible for extracting emotional features in the original audio sequence, and the style encoder is responsible for extracting features related to personal style. The architecture of the three encoders follows the DeiT visual transformer, which is used to process the filter bank images extracted from the input audio, represented as , where a represents the filterbank feature of the original audio sequence, c, e, and s represent the content latent vector, emotion latent vector, and personal style latent vector, respectively. This module is pre-trained using the emotional speech dataset.

[0045] Latent Diffusion Model Module: The Latent Diffusion Model is a generative model based on the diffusion process, which is used to generate samples of complex data distributions, such as images, audio, or action sequences. This model gradually adds noise data into random noise in a high-dimensional space, and then learns the inverse process, that is, how to recover the original data from the noise.

[0046] The forward diffusion process is a process of gradually adding noise to the data. In this process, a noise scheduler with fixed variance and linear scaling is used. For the motion latent vector , adding noise in D diffusion time steps to obtain , as shown in the following formula:

[0047] ,

[0048] in, , , represents the variance of the diffusion process, represents random noise, represents the diffusion time step The motion latent vector of represents the motion latent vector without adding noise.

[0049] like Figure 2 As shown in FIG, the denoising process includes conditional denoising of a conditional motion noise latent vector to obtain a denoised motion latent vector. The structure of the denoiser is a UNet structure, as shown in the following formula: ,in is the motion latent vector, superscript It represents the diffusion time step Related, is the diffusion time step The sinusoidal position encoding of , c, e, s represent the content latent vector, emotion latent vector and personal style latent vector respectively.

[0050] In order to better capture the emotional and personal style features, thereby improving the expressiveness of the generated actions, the present invention improves the feature fusion mechanism in the diffusion model, adds the improvement of the cross-attention feature fusion mechanism, and proposes a hybrid cross-attention feature fusion mechanism, such as Figure 3 shown.

[0051] The hybrid cross-attention feature fusion mechanism introduces a cross-attention layer into the Transformer block of the original diffusion model. Specifically, the noise latent vector , diffusion time-stepped sinusoidal position encoding , the content latent vector c and the emotion latent vector e, and the personal style latent vector s are concatenated according to the first dimension of the vector to obtain a new noise latent vector ; Concatenate the emotion latent vector e and the personal style latent vector s according to the first dimension of the vector to obtain the conditional latent vector Context. and conditional vector Context as the input of the cross attention layer, and transform them into and Context are converted into query vector Q, key vector K and value vector V.

[0052] Then the dot product of the query vector Q and the key vector K is calculated to get the attention score and scale it. The softmax operation is then performed on the attention score to get the attention weight, which is used to weight the sum of the value vector. The result is restored to the original dimension and passed through a linear layer to get the final output. Finally, the output of the cross attention layer is added to the noise potential vector for subsequent propagation.

[0053] Human motion prior module: It is a Transformer architecture based on variational autoencoder (VAE) for encoding and decoding 3D human motion. Figure 4 As shown, the module consists of two parts: encoder and decoder .in and The structure is the same, both use a UNet-like structure with Transformer blocks as units. There are jump connections between the Transformer blocks, which can help the network recover more detailed information during the decoding stage. Position encoding PE (PositionEmbeddings) provides information about the position of each element in the action sequence, so as to better understand the order relationship of the sequence. And inject the action sequence into the decoder In each multi-head attention layer of .

[0054] Encoder Receive a series of T frames of action sequence As input, the action sequence Including facial expression sequences and gesture sequences , the output two important parameters are the mean vector and the covariance matrix Combining to form a normal distribution, using reparameterization to sample from the normal distribution, the input action sequence Encoded as the reconstructed motion latent vector , here represents the action sequence from time step 1 to time step T, including the facial expression sequence from time step 1 to time step T and gesture sequences .

[0055] Decoder Receive the predicted motion latent vector and the reconstructed motion latent vector as input, and input the predicted motion latent vector and the reconstructed motion latent vector as memory to each cross-attention Transformer layer, and use the information decoding of the latent space to decode the predicted motion latent vector to obtain the predicted action sequence , where the predicted action sequence Including predicted facial expression sequences and predicted gesture action sequences , and the reconstructed motion potential vector is decoded to obtain the reconstructed action sequence , where the reconstructed action sequence Including reconstructed facial expression sequences and reconstructed gesture action sequences .

[0056] Training module: Optimize the latent diffusion model module and the human motion prior module, and ensure that the original audio and motion latent vector can be aligned during the denoising process by calculating the diffusion model gradient and the loss function of the human motion prior module. The loss function includes reconstruction loss, KL divergence loss and alignment reconstruction loss.

[0057] Specifically, the stop motion prior encoder The gradient calculation of the motion potential vector is obtained by disabling the gradient , the motion latent vector with gradients disabled Generating motion noise latent vectors via a diffusion process , The concatenation of the conditional latent vector Context is used as the input of the denoiser ∆. At this stage, the noise predicted by the denoiser is obtained. Used to calculate diffusion model gradients.

[0058] By iteratively applying the denoiser ∆ from Get the predicted motion latent vector , and then through the motion prior decoder Generate predicted action sequences ,in Indicates that the gradient calculation is not taken into account.

[0059] The loss function used in the emotion-enhanced facial expression and gesture generation task consists of three parts: motion prior loss, alignment reconstruction loss, and denoising model supervision loss. The motion prior loss includes reconstruction loss and KL divergence loss.

[0060] To train the human motion prior module, the reconstruction loss is the standard VAE loss, including the pose parameters and vertex coordinates The reconstruction loss on , where the action sequence is passed through the SMPLX model to obtain the average mesh shape in three-dimensional space to obtain vertex coordinates, and L represents the smooth L1 metric, denoted as The pose parameter reconstruction loss measures the true action sequence and reconstruct action sequences The vertex coordinate reconstruction loss measures the true vertex coordinates and reconstruct the vertex coordinates The difference between the two is as follows:

[0061] .

[0062] KL divergence loss: KL divergence loss The difference between the variational distribution and the prior distribution is measured to make the variational distribution close to the prior distribution. The variational distribution is a normal distribution composed of the mean and covariance matrix output by the motion prior encoder, and the prior distribution is a standard normal distribution composed of 0 and 1. The specific formula is as follows:

[0063] ,in represents the mean of the variational distribution, represents the standard deviation of the variational distribution, represents the dimension of the latent space.

[0064] Aligned reconstruction loss: To ensure the alignment between the facial expression and hand gesture action sequences generated by the diffusion process and the input audio, an aligned reconstruction loss is applied to the predicted action pose parameters and vertex coordinates. Measured the real action sequence and predict action sequences The difference between the vertex coordinate alignment reconstruction loss Measure the real vertex coordinates and predicted vertex coordinates The difference between the two is shown below: .

[0065] Denoiser Supervision Loss: The purpose of the denoiser supervision loss is to ensure that the denoiser can effectively remove noise from the latent variable so that it matches the sampled noise vector as closely as possible. The calculation formula is as follows: ,in In the diffusion step From the standard normal distribution The sampled noise vector, is the latent variable In the diffusion time step Perform denoising.

[0066] The overall loss function is expressed as:

[0067] .

[0068] The inference process of the emotion-enhanced facial expression and gesture action generation network uses DDIM to infer high-quality action prediction samples through a small number of denoised time steps. During inference, a noisy latent vector is extracted from N(0,I) , and concatenated with the conditional latent vector as conditional noise. The denoiser is iteratively applied Generate the second motion latent vector, which is the motion latent vector after complete denoising , and finally through the motion prior decoder Decode it into the final action prediction sequence .

[0069] The rendering image generation module is used to fit the facial expression coefficients and three-dimensional human body parameters in the aligned predicted action sequence and the reconstructed action sequence to form two-dimensional key points in the plane coordinates, and then generate a neural rendering image with facial expression and whole body semantic information.

[0070] like Figure 5 As shown, in the embodiment, the neural rendering image generation aims to fit the facial Faceverse expression coefficients and SMPLX three-dimensional human body parameters generated by voice drive into two-dimensional key points of plane coordinates, and generate a neural rendering image with facial and whole body semantic information, and the specific steps are as follows:

[0071] The Faceverse expression coefficients generated by voice-driven are used to fit the 3D facial key points through the Faceverse 3DMM model. The SMPLX 3D human body parameters generated by voice-driven are used to fit the 3D human body key points through the SMPLX model. The camera parameters extracted from the first frame image are used to project the 3D key points onto the 2D plane using the camera projection matrix to obtain a 2D key point sequence.

[0072] Generate CCBR neural rendering images: CCBR (Color, Coordinate, Binary, Radius) images are represented as an 8-bit RGB image, which is used to represent the semantic information of facial and body posture. For body posture key points, each key point is drawn as a circle with a fixed radius, and each key point is given a predefined fixed value, where the values ​​of the red and green channels are obtained by the normalized UV coordinates, and the blue channel uses a predefined fixed value. For facial key points, based on their normalized coordinates, they are rendered using Nvdiffrast, a neural network-based rasterization technology.

[0073] like Figure 6 As shown, the embodiment also provides a method for driving and presenting a digital human with enhanced emotion, which is applied to the above system and includes the following steps:

[0074] Obtain the original audio sequence, decode it into content latent vector, emotion latent vector and personal style latent vector based on the audio decoupling module, combine the motion noise and diffusion time step encoding transformed by the latent diffusion model module, and iteratively denoise to obtain the predicted motion latent vector;

[0075] Encode the action sequences including the facial expression sequence and the gesture action sequence through the motion prior encoder to obtain the reconstructed motion latent vector; use the predicted motion latent vector and the reconstructed motion latent vector as the input of the motion prior decoder to decode and obtain the predicted action sequence and the reconstructed action sequence;

[0076] Based on the decoded predicted action sequence and the reconstructed action sequence, the denoiser supervision loss, reconstruction loss, KL divergence loss and alignment reconstruction loss are combined to align the motion latent vectors in the original audio sequence with the decoded predicted action sequence and the reconstructed action sequence;

[0077] The rendered image generation module is used to render the facial expression coefficients and three-dimensional human body parameters in the aligned predicted action sequence to generate a neural rendered image with facial and full-body semantic information, thus completing the driving and presentation of the digital human.

[0078] The present invention is extended to the task of synchronously generating facial expressions and hand gestures, achieving the integrity of digital human generated movements. By introducing the cross-attention feature fusion mechanism, the expression of emotional and style features can be effectively enhanced, so that the generated facial expressions and hand gestures are no longer mechanical and lack emotional color, but can better integrate the corresponding emotional elements, so that the facial expressions and hand gestures are consistent with the emotional state conveyed by the emotional voice. For example, when the emotional voice expresses joy, the facial expressions and hand gestures can show this joy through more vivid, lively and contagious movements (such as cheerful gestures, stretched facial expressions, etc.), which greatly enhances the emotional expressiveness of the digital human's facial expressions and hand gestures.

[0079] The specific implementation methods described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. An emotion-enhanced digital human driving and presentation system, characterized in that: The system includes an audio decoupling module, a potential diffusion model module, a human motion prior module, a training module, and a rendering image generation module; The audio decoupling module includes three independent encoders for decomposing the original audio sequence into three independent feature latent vectors, represented as a content latent vector, an emotion latent vector and a personal style latent vector; The latent diffusion model module is used to convert the original action sequence into motion noise in a high-dimensional space by adding noise, and then convert the content latent vector, the emotion latent vector, the personal style latent vector, the diffusion time step encoding and the motion noise into a predicted motion latent vector through iterative denoising based on a fusion mechanism; The human body motion prior module includes a motion prior encoder and a motion prior decoder, which are used for encoding and decoding of 3D human body motions, wherein the motion prior encoder is used to provide information about the position of each element in the motion sequence, and reconstructs the information about the position of each element in the motion sequence to generate a reconstructed motion potential vector; the motion prior decoder is used to receive the predicted motion potential vector and the reconstructed motion potential vector as input, and decode to obtain the predicted motion sequence and the reconstructed motion sequence; the motion sequence includes: a facial expression sequence or a gesture motion sequence; The training module is used to optimize the potential diffusion model module and the human motion prior module, and align the original audio with the motion potential vector by calculating the potential diffusion model gradient and the loss function of the human motion prior module, wherein the loss function of the human motion prior module includes reconstruction loss, KL divergence loss and alignment reconstruction loss; The rendering image generation module is used to form two-dimensional key points of plane coordinates according to the facial expression coefficients and three-dimensional human body parameters in the aligned predicted action sequence, and generate a neural rendering image with facial expression and whole body semantic information.

2. The emotion-enhanced digital human driving and presentation system according to claim 1, characterized in that: The three independent encoders include a content encoder, an emotion encoder and a personal style encoder; The content encoder is used to extract information related to speech rhythm and speech as a content latent vector; The emotion encoder is used to extract the emotion features in the original audio sequence as the emotion latent vector; The personal emotion encoder is used to extract features related to personal style as a personal style latent vector.

3. The emotion-enhanced digital human driving and presentation system according to claim 1, characterized in that: The fusion-based mechanism converts the content latent vector, the emotion latent vector, the personal style latent vector, the diffusion time step coding and the motion noise into a predicted motion latent vector through iterative denoising, including: introducing a cross-attention layer in the Transformer block, concatenating the content latent vector, the emotion latent vector, the personal style latent vector, the diffusion time step coding and the motion noise according to the first dimension of the vector to obtain a new noise latent vector, concatenating the emotion latent vector and the personal style latent vector according to the first dimension of the vector to obtain a conditional latent vector, using the new noise latent vector and the conditional latent vector as inputs of the cross-attention layer, obtaining the final output through a linear layer, and iterative denoising to obtain a predicted motion latent vector for subsequent propagation.

4. The emotion-enhanced digital human driving and presentation system according to claim 3, characterized in that: The Transformer block includes two independent Transformer blocks, namely a facial expression Transformer block and a gesture action Transformer block, which are used to receive facial expression noise and gesture action noise for denoising.

5. The emotion-enhanced digital human driving and presentation system according to claim 3, characterized in that: In the iterative denoising process, a denoiser supervision loss function is introduced to remove noise from the motion latent vector so that the motion latent vector matches the motion noise.

6. The emotion-enhanced digital human driving and presentation system according to claim 1, characterized in that: The motion prior encoder and the motion prior decoder have the same structure, both using a UNet structure with Transformer blocks as units, and there are jump connections between Transformer blocks to recover more detail information in the decoding stage.

7. The emotion-enhanced digital human driving and presentation system according to claim 1, characterized in that: The method of reconstructing and encoding the information of the position of each element in the action sequence to generate a reconstructed motion latent vector includes: a motion prior encoder receives an action sequence of a facial expression sequence and a gesture action sequence as input, outputs a normal distribution composed of a mean and a covariance matrix, and uses reparameterization to sample from the normal distribution to generate a reconstructed motion latent vector.

8. The emotion-enhanced digital human driving and presentation system according to claim 1, characterized in that: The reconstruction loss includes posture parameter reconstruction loss and vertex coordinate reconstruction loss, wherein the posture parameter reconstruction loss is used to measure the difference between the real action sequence and the reconstructed action sequence, and the vertex coordinate reconstruction loss is used to measure the difference between the real vertex coordinates and the reconstructed vertex coordinates, wherein the action sequence is obtained through the SMPLX model to obtain the average mesh shape in three-dimensional space to obtain the vertex coordinates; The KL divergence loss is used to measure the difference between the variational distribution and the prior distribution, so that the variational distribution is close to the prior distribution, wherein the variational distribution is a normal distribution composed of the mean and covariance matrix output by the motion prior encoder, and the prior distribution is a standard normal distribution composed of 0 and 1; The alignment reconstruction loss includes posture parameter alignment reconstruction loss and vertex coordinate alignment reconstruction loss, wherein the posture parameter alignment reconstruction loss is used to measure the difference between the real action sequence and the predicted action sequence, and the vertex coordinate alignment reconstruction loss is used to measure the difference between the real vertex coordinates and the predicted vertex coordinates.

9. A method for driving and presenting a digital human with enhanced emotion, characterized in that: The method uses the emotion-enhanced digital human driving and presentation system according to any one of claims 1 to 8, comprising the following steps: Obtain the original audio sequence, decouple the original audio sequence into content latent vector, emotion latent vector and personal style latent vector based on the audio decoupling module, combine the motion noise and diffusion time step encoding transformed by the latent diffusion model module, and iteratively denoise to obtain the predicted motion latent vector; Encode the action sequences including the facial expression sequence and the gesture action sequence through the motion prior encoder to obtain the reconstructed motion latent vector; use the predicted motion latent vector and the reconstructed motion latent vector as the input of the motion prior decoder to decode and obtain the predicted action sequence and the reconstructed action sequence; Based on the decoded predicted action sequence and the reconstructed action sequence, the denoiser supervision loss, reconstruction loss, KL divergence loss and alignment reconstruction loss are combined to align the motion latent vectors in the original audio sequence with the decoded predicted action sequence and the reconstructed action sequence; The rendered image generation module is used to render the facial expression coefficients and three-dimensional human body parameters in the aligned predicted action sequence to generate a neural rendered image with facial and full-body semantic information, thus completing the driving and presentation of the digital human.

Images