A real-time generation method and device for digital characters

Abstract

The present application discloses a real-time generation method and device for a digital character, which obtains audio content and a two-dimensional digital character image, processes the audio content through a pre-trained full-body motion generation model to obtain a three-dimensional full-body posture sequence, maps the three-dimensional full-body posture sequence into a two-dimensional full-body posture sequence, performs real-time posture driving on the two-dimensional full-body posture sequence and the two-dimensional digital character image through a pre-trained real-time posture drive generation model to obtain a corresponding posture video, and adjusts the facial image of the digital character in the corresponding posture video in real time through a pre-built lip synchronization model and audio content to make the lip shape in the facial image consistent with the audio content to obtain a target digital character video, wherein the target digital character video is a two-dimensional digital character video with consistent semantics of full-body motion, facial expression and voice content.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and more specifically, to a method and device for real-time generation of digital characters. Background Art

[0002] Two-dimensional digital human generation technology is a method of using computer vision technology to create digital human images that are close to human images. Two-dimensional digital human generation technology aims to drive two-dimensional digital characters to perform the same behaviors as humans by transplanting human movements and expressions. It has a wide range of applications in live broadcasting, e-commerce, virtual social networking and other fields.

[0003] However, the existing two-dimensional digital character generation movements are relatively monotonous and the content expression is single. Many of them are driven by preset movements and audio drivers. During the digital character driving process, the digital character's expression and movement are driven separately, which consumes a lot of resources and is difficult to meet real-time application needs, thereby reducing the interactive experience with users.

[0004] Therefore, how to improve the interactivity and authenticity between digital characters and users is an urgent problem to be solved in this application. Summary of the invention

[0005] In view of this, the present application discloses a real-time generation method and device for a digital character, aiming to obtain a two-dimensional digital character video with semantically consistent whole-body movements, facial expressions, and voice content, thereby improving the interactivity and authenticity of the digital character with the user.

[0006] In order to achieve the above purpose, the disclosed technical solution is as follows:

[0007] The first aspect of the present application discloses a real-time generation method of a digital character, the method comprising:

[0008] Acquiring audio content and two-dimensional digital character image;

[0009] The audio content is processed by a pre-trained full-body motion generation model to obtain a three-dimensional full-body posture sequence; wherein the full-body motion generation model is obtained by jointly training the facial motion expression training data and the full-body limb motion training data through audio training data;

[0010] Mapping the three-dimensional whole-body posture sequence into a two-dimensional whole-body posture sequence;

[0011] By using a pre-trained real-time posture drive generation model, the two-dimensional full-body posture sequence and the two-dimensional digital human image are subjected to real-time posture drive to obtain a corresponding posture video;

[0012] Through the pre-built lip synchronization model and the audio content, the facial image of the digital character in the corresponding posture video is adjusted in real time so that the lip shape in the facial image is consistent with the audio content, thereby obtaining a target digital character video; wherein the target digital character video is a two-dimensional digital character video with consistent semantics of full body movements, facial expressions and voice content.

[0013] Preferably, the process of training the whole body motion generation model includes:

[0014] Acquire training data; the training data at least includes audio training data, whole body movement training data and facial movement expression training data;

[0015] Extracting Mel spectrum low-level features, high-level features and speech text semantic features from the audio training data; the Mel spectrum low-level features represent the spectrum information of the speech signal; the high-level features represent the semantic information of the speech signal;

[0016] Inputting the Mel spectrum low-level features, the high-level features and the speech text semantic features into the action speech fusion sharing module to obtain fusion features;

[0017] Inputting the facial action expression training data, the facial expression random noise and the fusion feature into an action and speech fusion sharing module to obtain a first fusion feature;

[0018] Inputting the first fused feature into an expression encoder for denoising training to predict facial expression noise training data;

[0019] Inputting the whole body limb movement training data, limb movement random noise and the fusion feature into the action and speech fusion sharing module to obtain a second fusion feature;

[0020] Inputting the second fused feature into a posture encoder for denoising training to predict limb noise training data;

[0021] The whole body motion generation model is trained by using the facial expression noise training data and the body noise training data until the whole body motion generation model reaches convergence.

[0022] Preferably, the three-dimensional whole-body posture sequence at least includes limb skeleton information and expression coordinate information, and mapping the three-dimensional whole-body posture sequence into a two-dimensional whole-body posture sequence includes:

[0023] Mapping the limb skeleton information and the expression coordinate information into three-dimensional whole body posture sequence coordinates through a preset mapping method;

[0024] The three-dimensional whole-body posture sequence coordinates are converted into a two-dimensional whole-body posture sequence.

[0025] Preferably, the real-time gesture driven generation model includes at least an appearance control network, a potential diffusion model and a gesture control module, and the process of training the real-time gesture driven generation model includes:

[0026] Get the distillation model weights;

[0027] Initializing a latent diffusion model according to the distillation model weights, and freezing the initialized latent diffusion model;

[0028] Training the appearance control network using two-dimensional digital human image training data;

[0029] Training the posture control module with random noise training data to achieve joint fine-tuning of the appearance control network and the posture control network, and returning to the step of training the appearance control network with two-dimensional digital human image training data until the latent diffusion model converges;

[0030] The converged latent diffusion model is accelerated by quantizing the model to complete the training process of the real-time posture-driven generation model.

[0031] Preferably, the pre-trained real-time posture drive generation model is used to perform real-time posture drive on the two-dimensional whole-body posture sequence and the two-dimensional digital human image to obtain a corresponding posture video, including:

[0032] Performing posture encoding on the two-dimensional whole-body posture sequence by means of a posture control module in a pre-trained real-time posture-driven generation model;

[0033] Performing appearance encoding on the two-dimensional digital human image by driving an appearance control network in a real-time posture-generated model;

[0034] Through the latent diffusion model in the real-time posture-driven generation model and the pre-added random noise, the two-dimensional full-body posture sequence after posture encoding and the two-dimensional digital human image after appearance encoding are diffused and denoised to obtain the corresponding posture video.

[0035] Preferably, the step of adjusting the facial image of the digital character in the corresponding posture video in real time by using the pre-built lip synchronization model and the audio content so that the lip shape in the facial image is consistent with the audio content to obtain the target digital character video includes:

[0036] Encode the facial image of the digital person in the corresponding pose video through a variational autoencoder frozen in a pre-built lip sync model;

[0037] encoding the audio content through a speech recognition model frozen in a lip synchronization model;

[0038] Through the lip synchronization model in the lip synchronization model, the encoded audio content is used to adjust the encoded facial image in real time, so that the lip shape in the facial image is consistent with the audio content, and the target digital character video is output through the generation network in the lip synchronization model.

[0039] The second aspect of the present application discloses a real-time generation device for a digital character, the device comprising:

[0040] An acquisition unit, used for acquiring audio content and a two-dimensional digital character image;

[0041] A processing unit, used to process the audio content through a pre-trained whole-body action generation model to obtain a three-dimensional whole-body posture sequence; wherein the whole-body action generation model is obtained by jointly training the facial action expression training data and the whole-body limb action training data guided by the audio training data;

[0042] A mapping unit, used for mapping the three-dimensional whole-body posture sequence into a two-dimensional whole-body posture sequence;

[0043] A real-time posture driving unit, used to perform real-time posture driving on the two-dimensional whole-body posture sequence and the two-dimensional digital human image through a pre-trained real-time posture driving generation model to obtain a corresponding posture video;

[0044] The adjustment unit is used to adjust the facial image of the digital character in the corresponding posture video in real time through the pre-built lip synchronization model and the audio content, so that the lip shape in the facial image is consistent with the audio content, and the target digital character video is obtained; wherein the target digital character video is a two-dimensional digital character video with consistent full body movements, facial expressions and voice content semantics.

[0045] Preferably, the processing unit for training the whole body motion generation model comprises:

[0046] A first acquisition module is used to acquire training data; the training data at least includes audio training data, whole body movement training data and facial movement expression training data;

[0047] An extraction module, used to extract Mel spectrum low-level features, high-level features and speech text semantic features from the audio training data; the Mel spectrum low-level features represent the spectrum information of the speech signal; the high-level features represent the semantic information of the speech signal;

[0048] A first input module is used to input the Mel spectrum low-level features, the high-level features and the speech text semantic features into the action speech fusion sharing module to obtain fusion features;

[0049] A second input module is used to input the facial action expression training data, facial expression random noise and the fusion feature into the action and speech fusion sharing module to obtain a first fusion feature;

[0050] A first denoising training module, used for inputting the first fusion feature into an expression encoder for denoising training to predict facial expression noise training data;

[0051] A third input module is used to input the whole body limb movement training data, limb movement random noise and the fusion feature into the action and speech fusion sharing module to obtain a second fusion feature;

[0052] A second denoising training module, used for inputting the second fused features into a posture encoder for denoising training to predict limb noise training data;

[0053] The first training module is used to train the whole body motion generation model through the facial expression noise training data and the limb noise training data until the whole body motion generation model reaches convergence.

[0054] Preferably, the three-dimensional whole-body posture sequence includes at least limb skeleton information and expression coordinate information, and the mapping unit includes:

[0055] A mapping module, used to map the limb skeleton information and the expression coordinate information into three-dimensional whole body posture sequence coordinates through a preset mapping method;

[0056] The conversion module is used to convert the three-dimensional whole-body posture sequence coordinates into a two-dimensional whole-body posture sequence.

[0057] Preferably, the real-time posture drive generation model includes an appearance control network, a potential diffusion model and a posture control module, and the real-time posture drive unit for training the real-time posture drive generation model includes:

[0058] The second acquisition module is used to obtain the distillation model weight;

[0059] An initialization module, used to initialize the potential diffusion model according to the distillation model weights, and freeze the initialized potential diffusion model;

[0060] A second training module, used to train the appearance control network using two-dimensional digital character image training data;

[0061] A third training module is used to train the posture control module through random noise training data to achieve joint fine-tuning of the appearance control network and the posture control network, and return to the step of training the appearance control network through the two-dimensional digital human image training data until the potential diffusion model converges;

[0062] The acceleration module is used to accelerate the converged potential diffusion model through the quantization model to complete the training process of the real-time posture-driven generation model.

[0063] It can be known from the above technical scheme that the present application discloses a real-time generation method and device for digital characters, which obtain audio content and a two-dimensional digital character image, and process the audio content through a pre-trained full-body action generation model to obtain a three-dimensional full-body posture sequence, wherein the full-body action generation model is obtained by jointly training facial action expression training data and full-body limb action training data through audio training data, and the three-dimensional full-body posture sequence is mapped into a two-dimensional full-body posture sequence, and a pre-trained real-time posture-driven generation model is used to perform real-time posture driving on the two-dimensional full-body posture sequence and the two-dimensional digital character image to obtain a corresponding posture video, and the facial image of the digital character in the corresponding posture video is adjusted in real time through a pre-constructed lip synchronization model and audio content, so that the lip shape in the facial image is consistent with the audio content, and a target digital character video is obtained, wherein the target digital character video is a two-dimensional digital character video with consistent semantics of full-body action, facial expression and voice content.

[0064] Through the above scheme, by pre-building a full-body motion generation model driven by audio training data to generate facial expressions and full-body limb movements, realistic facial expressions are generated based on the joint representation of facial expressions and full-body postures. While generating realistic facial expressions, full-body movements coordinated with facial expressions are generated based on audio content. The real-time posture-driven generation model is used to perform real-time posture driving on the two-dimensional full-body posture sequence and the two-dimensional digital character image, thereby generating a corresponding posture video that is consistent with the character movement and facial expression in the reference two-dimensional digital character image, and the facial image of the digital character in the corresponding posture video is adjusted in real time through the lip synchronization model and audio content, so that the lip shape in the facial image is consistent with the audio content, and a two-dimensional digital character video with consistent semantics of full-body movements, facial expressions and voice content is obtained, thereby improving the interactivity and authenticity of the digital character and the user. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0066] Figure 1 A flowchart of a method for real-time generation of a digital character disclosed in an embodiment of the present application;

[0067] Figure 2A schematic diagram of a training whole-body motion generation model disclosed in an embodiment of the present application;

[0068] Figure 3 A schematic diagram of generating a corresponding posture video disclosed in an embodiment of the present application;

[0069] Figure 4 A schematic diagram of another method for real-time generation of a digital character disclosed in an embodiment of the present application;

[0070] Figure 5 A schematic diagram of the structure of a real-time digital character generation device disclosed in an embodiment of the present application;

[0071] Figure 6 This is a schematic diagram of the structure of an electronic device disclosed in an embodiment of the present application. DETAILED DESCRIPTION

[0072] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0073] In this application, the terms "comprises", "comprising" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprising a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element.

[0074] As can be seen from the background technology, the existing two-dimensional digital character generation movements are relatively monotonous, and the content expression is single. Many of them use preset movements and audio drivers to drive digital characters. Since the digital character expressions and movements are driven separately during the digital character driving process, the resource consumption is large, and it is difficult to meet the real-time application requirements, thereby reducing the interactive experience with users. Therefore, how to improve the interactivity and authenticity of digital characters and users is an urgent problem to be solved in this application.

[0075] In order to solve the above problems, the present application discloses a real-time generation method and device for digital characters, which pre-builds a full-body motion generation model driven by audio training data to generate facial expressions and full-body limb movements, and achieves the generation of realistic facial expressions based on the joint representation of facial expressions and full-body postures. While generating realistic facial expressions, full-body movements coordinated with facial expressions are generated based on audio content. The real-time posture-driven generation model is used to perform real-time posture driving on a two-dimensional full-body posture sequence and a two-dimensional digital character image, thereby generating a corresponding posture video that is consistent with the character movements and facial expressions in the reference two-dimensional digital character image, and the facial image of the digital character in the corresponding posture video is adjusted in real time through a lip synchronization model and audio content, so that the lip shape in the facial image is consistent with the audio content, and a two-dimensional digital character video with semantically consistent full-body movements, facial expressions and voice content is obtained, thereby improving the interactivity and authenticity of the digital character and the user. The specific implementation method is specifically described through the following embodiments.

[0076] refer to Figure 1 As shown in the first embodiment of the present application, a real-time generation method of a digital character is disclosed, and the real-time generation method of a digital character mainly includes the following steps:

[0077] S101: Acquire audio content and a two-dimensional digital character image.

[0078] In S101 , audio content and a two-dimensional digital character image are externally input non-training data for generating a digital character.

[0079] S102: Processing the audio content through a pre-trained full-body motion generation model to obtain a three-dimensional full-body posture sequence; wherein the full-body motion generation model is obtained by jointly training the facial motion expression training data and the full-body limb motion training data through audio training data.

[0080] In S102, the audio content is input into a trained whole-body motion generation model to obtain a three-dimensional whole-body posture sequence, wherein the three-dimensional whole-body posture sequence includes information such as limb skeleton information and expression coordinate information. Since the three-dimensional whole-body posture sequence is based on a 3D format, it is necessary to map the three-dimensional whole-body posture sequence into a three-dimensional coordinate form, and convert the three-dimensional whole-body posture sequence in the three-dimensional coordinate form into a two-dimensional whole-body posture sequence.

[0081] The specific process of training the whole body action generation model, such as Figure 2 and A1-A8.

[0082] Figure 2 , obtain audio training data, body movement training data, and facial movement and expression training data (in beat-smplex-flame format);

[0083] Extract Mel-Spectrogram low-level features (spectral information of speech signals) A from audio training data, use HuBERT to extract high-level features B (high-level semantic information of speech signals) from audio training data, and use whisperencoder to extract corresponding speech-text semantic features C from audio training data, that is, use encoder encoder to extract speech features;

[0084] The corresponding body movement training data, facial movement expression training data, low-level features (A), high-level features (B) and speech-text semantic features (C) are fused (concatenated) by channel to obtain fused features (E);

[0085] The fusion features, facial expression random noise and facial action expression training data are sent to the expression encoder module in the action speech fusion shared module based on the deep learning model (transformer) for denoising training to obtain the predicted facial expression noise training data;

[0086] The fusion features, limb movement random noise and whole body limb movement training data are sent to the Transformer-based gesture encoder module for denoising training to obtain the predicted limb noise training data;

[0087] The whole body motion generation model is trained according to the predicted facial expression noise training data and the predicted limb noise training data until the whole body motion generation model reaches convergence.

[0088] A1: Obtain training data; the training data at least includes audio training data, whole body movement training data, and facial movement expression training data.

[0089] Among them, the format of facial expression training data can be a full-body dataset (beat-smplex-flame) format of speech and grid-based motion data, etc. The beat-smplex-flame format contains high-quality 3D motion capture datasets with parameters such as MoShed body (SMPLX) and head (FLAME).

[0090] Fundamentally different from existing joint training methods (such as face-body scan data modeling), this scheme uses audio data to guide the joint training of facial expression and body movements, and the generated movement-expression sequences are richer.

[0091] A2: Extract Mel-Spectrogram low-level features, high-level features and speech text semantic features from audio training data; Mel-Spectrogram low-level features represent the spectral information of speech signals; high-level features represent the semantic information of speech signals.

[0092] Among them, the low-level features of Mel-Spectrogram include pitch, duration, intensity, sound quality, intonation and other information.

[0093] The semantic information of speech signals, that is, the high-level semantic information of speech signals. High-level semantic information refers to the deep semantics in the text, including more abstract and complex information such as logical reasoning, semantic association, and semantic role. Through high-level semantic information, it is easy to get the tone, voice, and action connection of the audio. It goes beyond the literal meaning of words and sentences. Through high-level semantic information, we can deeply understand the background, purpose, and context of the text.

[0094] For example, extract the low-level Mel-Spectrogram feature A (low-level) from the audio training data, and use the deep speech model (HuBERT) to extract the high-level feature B, so as to obtain the connection between the audio's intonation, tone, and action. Then use the speech recognition model (whisper encoder) to extract the speech text semantic feature C corresponding to the connection between the audio's intonation, tone, and action, so as to fully understand the semantics.

[0095] A3: Input the Mel-Spectrogram low-level features, high-level features and speech-text semantic features into the Motion-Speech Fusion Module to obtain the fusion features (E).

[0096] In A3, in order to make full use of the semantic information of low-level audio features (pitch, duration, intensity, sound quality, intonation, etc.) and high-level features, the obtained low-level features and high-level features are input into the action-speech fusion sharing module by channel for feature fusion to obtain fused features.

[0097] It should be noted that the action-speech fusion sharing module does not adopt the traditional cross-attention mechanism, but directly fuses the Mel-Spectrogram low-level features, high-level features and speech-text semantic features according to channels. The advantage of this is that the audio and action features are better aligned in the time dimension.

[0098] A4: Input the facial action expression training data, facial expression random noise and fusion features into the action speech fusion sharing module to obtain the first fusion feature.

[0099] In A4, the action-speech fusion shared module contains a normalization layer (LayerNorm, LN) and a multilayer perceptron (Multilayer Perceptron, MLP) layer, which helps to ensure numerical stability and fast model convergence.

[0100] Among them, the action-speech fusion sharing module inputs the given facial expression data, noise and fusion features into the action-speech fusion sharing module, and then outputs the facial expression-whole body action-speech fusion features, that is, the first fusion features.

[0101] A5: The first fused feature is input into an expression-encoder based on a deep learning model (transformer) for denoising training to predict facial expression noise training data.

[0102] A6: Input the whole body limb movement training data, limb movement random noise and fusion features into the action and speech fusion sharing module to obtain the second fusion feature.

[0103] In A6, the whole body limb movement training data, limb movement random noise and fusion features are processed through the action and speech fusion sharing module to output the action and speech fusion features, that is, the second fusion features.

[0104] A7: Input the second fused feature into the posture encoder for denoising training to predict the limb noise training data.

[0105] In A7, the second fusion feature is input into the gesture-encoder based on the deep learning model (transformer) through the action-speech fusion shared module for denoising training to predict the body noise training data.

[0106] A8: The whole body motion generation model is trained using facial expression noise training data and limb noise training data until the whole body motion generation model reaches convergence.

[0107] The action-speech fusion shared module uses a robust regression loss function (huber loss) and a velocity loss function (velocity loss). Huber loss and velocity loss are mainly used to improve the performance of the full-body action generation model and optimize the training process, and ultimately obtain a full-body action generation model that generates expressions and full-body postures driven by real-time audio. After the full-body action generation model reaches convergence, it can be reused without the need for retraining.

[0108] S103: Mapping the three-dimensional whole-body posture sequence into a two-dimensional whole-body posture sequence.

[0109] In S103, the limb skeleton information and the expression coordinate information are mapped into three-dimensional whole body posture sequence coordinates through a preset mapping method, and the three-dimensional whole body posture sequence coordinates are converted into a two-dimensional whole body posture sequence.

[0110] It should be noted that the preset mapping method includes but is not limited to the projection matrix method.

[0111] S104: Performing real-time posture driving on the two-dimensional whole-body posture sequence and the two-dimensional digital human image through a pre-trained real-time posture driving generation model to obtain a corresponding posture video.

[0112] In S104, a two-dimensional full-body posture sequence and a two-dimensional digital human image (any desired image can be customized) are input into a trained single-step diffusion real-time posture-driven generation model (distillation model Hyper-SD) to obtain a digital human video with the corresponding posture.

[0113] A real-time posture-driven generation model is constructed based on a single-step diffusion generation video algorithm framework. Compared with other existing posture-driven algorithms, this real-time posture-driven generation model can quickly generate posture videos in a single step and has strong generalization capabilities in complex movements.

[0114] The real-time pose-driven generation model consists of an appearance control network, a latent diffusion model (SD-UNet), and a pose control module.

[0115] The posture control module is a UNet network structure, which is used to encode the posture of the two-dimensional whole body posture sequence. The appearance control network is the same as SD-UNet, which is used to encode the appearance of the two-dimensional digital character image.

[0116] Specifically, the corresponding posture video is obtained as follows Figure 3 shown.

[0117] Figure 3 In the method, first, the two-dimensional whole-body posture sequence is sent to a posture control module in a pre-built real-time posture-driven generation model, and the posture control module performs posture encoding on the two-dimensional whole-body posture sequence;

[0118] Then, the appearance of the two-dimensional digital human image is encoded through the appearance control network in the real-time pose-driven generation model;

[0119] Finally, through the latent diffusion model in the real-time posture-driven generation model and the pre-added random noise, the two-dimensional full-body posture sequence after posture encoding and the two-dimensional digital character image after appearance encoding are diffused and denoised to obtain the corresponding posture video.

[0120] The specific process of training the real-time posture-driven generation model is shown in B1-B5.

[0121] B1: Get the distillation model (Hyper-SD) weights.

[0122] It should be noted that the traditional posture-driven generation model has a relatively slow generation speed and cannot achieve the desired speed. After testing, it is found that the time is mainly spent on SD-UNet denoising. The denoising module often requires multiple steps to achieve satisfactory results, which limits its efficiency in practical applications. Therefore, in response to this problem, this solution uses Hyper-SD weights to initialize SD-UNet. Since Hyper-SD uses a technology called trajectory segmented consistency distillation (TSCD), it can generate high-quality images in one step, which can greatly reduce the inference time compared to existing models. Considering the problem of model feature matching, the model needs to be re-pretrained. Finally, both the generation quality and generation speed can be greatly improved.

[0123] The TSCD technique divides the entire training process of the diffusion model into k segments in the entire time step range [0, T], where T=24; k=8, 4, 2, 1. It gradually performs piecewise consistent model distillation to refine multiple stages of the training process, and performs consistent distillation on the model in each stage to ensure that the model can maintain consistent generation quality at different stages. In single-step reasoning, its effect even surpasses the original model and greatly reduces the reasoning speed.

[0124] B2: Initialize SD-UNet according to the distilled model weights and freeze the initialized latent SD-UNet.

[0125] B3: Train the appearance control network using two-dimensional digital character image training data.

[0126] B4: Train the posture control module through random noise training data to jointly fine-tune the appearance control network and the posture control network, and return to the step of training the appearance control network through 2D digital character training pictures until SD-UNet converges.

[0127] In B4, loop B2 to B3 until the model converges. For joint fine-tuning, the batch size can be set to 192, the training steps can be set to 10000, the resolution is 512×512, and the learning rate is 0.0001.

[0128] B5: Accelerate the converged latent diffusion model by quantizing the model to complete the training process of the real-time gesture-driven generation model.

[0129] In B5, 8-bit quantization is performed using the model optimizer (NVIDIA TensorRT) to further accelerate SD-UNet to obtain the final real-time gesture-driven generation model. The quantization steps are as follows:

[0130] First, use the NVIDIA TensorRT toolkit to convert it into a quantized model (INT8), that is, an INT8 quantized model;

[0131] Then, convert the INT8 quantized model into an INT8 UNet ONNX model;

[0132] Finally, the inference engine (TensorRT) was created using the INT8 UNet ONNX model, completing the quantization deployment of the Hyper-SD-based diffusion model-based framework (magicpose) model training and deep learning inference optimization tool (tensorrt).

[0133] Use the 2D full-body posture sequence and the 2D digital human image (any customized digital human image or real person image can be used) to input into the above quantized real-time posture-driven generation model, set the step to 1, and perform single-step diffusion to obtain the digital human video of the corresponding posture.

[0134] S105: Using a pre-built lip synchronization model and audio content, adjust the facial image of the digital character in the corresponding posture video in real time so that the lip shape in the facial image is consistent with the audio content, thereby obtaining a target digital character video; wherein the target digital character video is a two-dimensional digital character video with consistent semantics of full body movements, facial expressions, and voice content.

[0135] The specific process of obtaining the target digital character video is shown in C1-C3.

[0136] C1: Encoding the facial image of a digital person in a pose video via a variational autoencoder (VEA) frozen in a pre-built lip sync model.

[0137] C2: Encode the audio content through the speech recognition model (whisper-tiny) frozen in the lip sync model.

[0138] C3: Through the lip synchronization (musetalk) model in the lip synchronization model, the encoded audio content is used to adjust the encoded facial image in real time, so that the lip shape in the facial image is consistent with the audio content, and the target digital character video is output through the generation network in the lip synchronization model.

[0139] Among them, the musetalk model is faster and has higher generation quality than existing lip sync models, such as the wave2lip model. The musetalk model is trained in latent space, where images are encoded by frozen VAEs and audio is encoded by frozen whisper-tiny models. The architecture of the generative network is borrowed from the UNet of stable-diffusion-v1-4, where audio embeddings are fused with image embeddings through cross-attention.

[0140] Although the architecture used by the musetalk model is very similar to Stable Diffusion, the musetalk model is unique in that it is not a diffusion model. Instead, the musetalk model operates by single-step repair in the latent space, so the musetalk model can drive the lips in real time and can automatically adjust the facial image of the digital character according to the input audio signal so that its lip shape is highly synchronized with the audio content. The encoded audio content and the encoded facial image are input into the musetalk model, and the musetalk model adjusts the encoded audio content to the encoded facial image in real time so that the lip shape in the facial image is consistent with the audio content, thereby outputting a target digital character video with consistent full-body movements, expressions, and speech content semantics.

[0141] In order to facilitate the understanding of the real-time generation process of digital characters, combined with Figure 4 Provide explanation. Figure 4 An overall schematic diagram of another method for real-time generation of digital characters is shown.

[0142] Figure 4 In , the audio content is input into a pre-trained full-body motion generation model;

[0143] The audio content is processed through a pre-trained full-body motion generation model to obtain a 3D full-body posture sequence;

[0144] Mapping a three-dimensional full-body pose sequence into a two-dimensional full-body pose sequence;

[0145] Inputting the two-dimensional full-body posture sequence and the acquired two-dimensional digital human image into the real-time posture-driven generation model;

[0146] Through the real-time posture-driven generation model, the 2D whole-body posture sequence and the 2D digital human image are driven in real time to obtain the digital human video with the corresponding posture;

[0147] The digital character video with the corresponding posture is input into the lip synchronization model to obtain a two-dimensional digital character video with consistent full body movements, facial expressions and speech content.

[0148] The beneficial effects of this application include the following:

[0149] Cost-effectiveness: Audio is input into the full-body motion generation model, and the pose sequence is directly output end-to-end. There is no need to add an additional expression transfer module, which saves costs. It only needs to be trained once, and the model can be reused for subsequent digital human production;

[0150] High efficiency: All modules can be processed in real time and respond quickly;

[0151] User experience: The full-body motion generation and real-time performance of digital characters greatly enhance the audience’s interactive experience and sense of reality;

[0152] Innovation: The application's integrated real-time driving solution for audio, expression, and whole-body motion and the introduction of a posture-driven generation module based on diffusion model distillation are both innovative.

[0153] The purpose of this application improvement is:

[0154] In the prior art, the movements of two-dimensional digital characters are relatively monotonous, the content expression is single, and the driving relationship between human voice intonation and facial expression is not fully utilized. In addition, many of them use preset movements to drive digital people. Even for audio-driven digital characters, it is only performed on the upper body, and the lower body is basically still. Some even only drive the facial movements of the digital person, and the body movements are fixed, which cannot meet the needs of full-body drive. However, the present application constructs a real-time audio-driven facial expression and full-body movement generation system, which uses facial expressions and postures to jointly represent, and can generate realistic facial expressions while generating coordinated full-body movements based on audio input. This solves the problem of incoordination when audio, facial and body movements are mixed.

[0155] A real-time posture-driven image-generated video system is constructed based on a real-time posture-driven generation model of distilled single-step diffusion. It can generate videos that are consistent with the character movements and expressions in the reference image based on the posture sequence. For example, posture can be used as a control signal to convert still images of characters into smooth corresponding action videos. In addition, the system uses a diffusion-based method to generate video frames, and gradually optimizes the output to achieve high-quality effects. In order to ultimately generate the corresponding pictures in real time, the present application also uses a Hyper-SD efficient diffusion distillation model to achieve image generation in 1 step. Ultimately, a good balance is achieved between effect and real-time.

[0156] The implementation of this application can improve the semantic coordination consistency and real-time performance of two-dimensional digital character movements and facial expressions, and the generated digital tasks are more natural, greatly improving the user's listening and speaking experience.

[0157] The beneficial effects of the first embodiment of the present application are as follows: by pre-building a full-body motion generation model generated by facial expressions and full-body limb movements driven by audio training data, realistic facial expressions are generated based on the joint representation of facial expressions and full-body postures, and while generating realistic facial expressions, full-body movements coordinated with facial expressions are generated based on audio content. The real-time posture-driven generation model is used to perform real-time posture driving on a two-dimensional full-body posture sequence and a two-dimensional digital character image, thereby generating a corresponding posture video that is consistent with the character movements and facial expressions in the reference two-dimensional digital character image, and the facial image of the digital character in the corresponding posture video is adjusted in real time through a lip synchronization model and audio content, so that the lip shape in the facial image is consistent with the audio content, and a two-dimensional digital character video with consistent semantics of full-body movements, facial expressions and voice content is obtained, thereby improving the interactivity and authenticity of the digital character with the user.

[0158] Based on the above Figure 1 The embodiment 1 of the present application discloses a method for generating a digital character in real time, and the embodiment 2 of the present application also discloses a device for generating a digital character in real time, such as Figure 5 As shown, the real-time generation device of the digital character mainly includes:

[0159] An acquisition unit 501 is used to acquire audio content and a two-dimensional digital character image;

[0160] The processing unit 502 is used to process the audio content through a pre-trained whole-body action generation model to obtain a three-dimensional whole-body posture sequence; wherein the whole-body action generation model is obtained by jointly training the audio training data, the facial action expression training data and the whole-body limb action training data;

[0161] A mapping unit 503, used to map the three-dimensional whole body posture sequence into a two-dimensional whole body posture sequence;

[0162] A real-time posture driving unit 504 is used to perform real-time posture driving on a two-dimensional whole-body posture sequence and a two-dimensional digital human image through a pre-trained real-time posture driving generation model to obtain a corresponding posture video;

[0163] The adjustment unit 505 is used to adjust the facial image of the digital character in the corresponding posture video in real time through the pre-built lip synchronization model and the audio content, so that the lip shape in the facial image is consistent with the audio content, and obtain the target digital character video; wherein the target digital character video is a two-dimensional digital character video with consistent full body movements, facial expressions and voice content semantics.

[0164] Furthermore, the processing unit 502 for training the whole body motion generation model includes:

[0165] The first acquisition module is used to acquire training data; the training data at least includes audio training data, whole body movement training data and facial movement expression training data;

[0166] An extraction module is used to extract Mel spectrum low-level features, high-level features and speech text semantic features from audio training data; Mel spectrum low-level features represent the spectrum information of speech signals; high-level features represent the semantic information of speech signals;

[0167] The first input module is used to input the low-level features of the Mel spectrum, the high-level features and the semantic features of the speech text into the action speech fusion sharing module to obtain the fusion features;

[0168] A second input module is used to input the facial action expression training data, the facial expression random noise and the fusion feature into the action speech fusion sharing module to obtain the first fusion feature;

[0169] A first denoising training module, used for inputting the first fusion feature into the expression encoder for denoising training to predict facial expression noise training data;

[0170] A third input module is used to input the whole body limb movement training data, limb movement random noise and fusion features into the action and speech fusion sharing module to obtain a second fusion feature;

[0171] A second denoising training module is used to input the second fused feature into the posture encoder for denoising training to predict limb noise training data;

[0172] The first training module is used to train the whole body motion generation model through facial expression noise training data and body noise training data until the whole body motion generation model reaches convergence.

[0173] Furthermore, the three-dimensional whole body posture sequence includes at least limb skeleton information and expression coordinate information, and the mapping unit 503 includes:

[0174] A mapping module, used to map the limb skeleton information and expression coordinate information into three-dimensional whole body posture sequence coordinates through a preset mapping method;

[0175] The conversion module is used to convert the three-dimensional whole-body pose sequence coordinates into a two-dimensional whole-body pose sequence.

[0176] Furthermore, the real-time gesture driven generation model includes an appearance control network, a potential diffusion model and a gesture control module, and the real-time gesture driving unit 504 for training the real-time gesture driven generation model includes:

[0177] The second acquisition module is used to obtain the distillation model weight;

[0178] An initialization module, used to initialize the latent diffusion model according to the distillation model weights and freeze the initialized latent diffusion model;

[0179] The second training module is used to train the appearance control network through two-dimensional digital character image training data;

[0180] A third training module is used to train the posture control module through random noise training data to achieve joint fine-tuning of the appearance control network and the posture control network, and return to the step of training the appearance control network through the two-dimensional digital human image training data until the potential diffusion model converges;

[0181] The acceleration module is used to accelerate the converged potential diffusion model through the quantization model to complete the training process of the real-time posture-driven generation model.

[0182] Furthermore, the real-time posture driving unit 504 includes:

[0183] A posture encoding module for posture encoding of a two-dimensional full-body posture sequence by using a posture control module in a pre-trained real-time posture-driven generation model;

[0184] An appearance encoding module, used to encode the appearance of a two-dimensional digital human image by driving the appearance control network in the real-time posture generation model;

[0185] The diffusion denoising module is used to perform diffusion denoising on the two-dimensional full-body posture sequence after posture encoding and the two-dimensional digital human image after appearance encoding through the potential diffusion model in the real-time posture-driven generation model and the pre-added random noise to obtain the corresponding posture video.

[0186] Further, the adjustment unit 505 includes:

[0187] A first encoding module, for encoding a facial image of a digital character in a corresponding pose video by a variational autoencoder frozen in a pre-built lip sync model;

[0188] A second encoding module, for encoding the audio content through a speech recognition model frozen in a lip synchronization model;

[0189] The real-time adjustment module is used to adjust the encoded audio content to the encoded facial image in real time through the lip synchronization model in the lip synchronization model, so that the lip shape in the facial image is consistent with the audio content, and output the target digital character video through the generation network in the lip synchronization model.

[0190] The beneficial effects of the second embodiment of the present application are as follows: by pre-building a full-body motion generation model generated by facial expressions and full-body limb movements driven by audio training data, realistic facial expressions are generated based on the joint representation of facial expressions and full-body postures, and while generating realistic facial expressions, full-body movements coordinated with facial expressions are generated based on audio content. The real-time posture-driven generation model is used to perform real-time posture driving on a two-dimensional full-body posture sequence and a two-dimensional digital character image, thereby generating a corresponding posture video that is consistent with the character movements and facial expressions in the reference two-dimensional digital character image, and the facial image of the digital character in the corresponding posture video is adjusted in real time through a lip synchronization model and audio content, so that the lip shape in the facial image is consistent with the audio content, and a two-dimensional digital character video with consistent semantics of full-body movements, facial expressions and voice content is obtained, thereby improving the interactivity and authenticity of the digital character with the user.

[0191] An embodiment of the present application further provides a storage medium, which includes stored instructions, wherein when the instructions are executed, the device where the storage medium is located is controlled to execute the real-time generation method of a digital character as described above.

[0192] The present application also provides an electronic device, whose structural diagram is as follows: Figure 6 As shown, it specifically includes a memory 601 and one or more instructions 602, wherein the one or more instructions 602 are stored in the memory 601 and are configured to be executed by one or more processors 603 to execute the one or more instructions 602 to perform the real-time generation method of the above-mentioned digital character.

[0193] For the aforementioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the order of the actions described, because according to the present application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0194] It should be noted that each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the embodiments can be referred to each other. For system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0195] The steps in the methods of the embodiments of the present application can be adjusted in order, combined, and deleted according to actual needs.

[0196] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are merely used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations.

[0197] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

[0198] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for real-time generation of a digital character, characterized in that: The method comprises: Acquiring audio content and two-dimensional digital character image; The audio content is processed by a pre-trained full-body motion generation model to obtain a three-dimensional full-body posture sequence; wherein the full-body motion generation model is obtained by jointly training the facial motion expression training data and the full-body limb motion training data through audio training data; Mapping the three-dimensional whole-body posture sequence into a two-dimensional whole-body posture sequence; By using a pre-trained real-time posture drive generation model, the two-dimensional full-body posture sequence and the two-dimensional digital human image are subjected to real-time posture drive to obtain a corresponding posture video; By using the pre-built lip synchronization model and the audio content, the facial image of the digital character in the corresponding posture video is adjusted in real time so that the lip shape in the facial image is consistent with the audio content, thereby obtaining a target digital character video; wherein the target digital character video is a two-dimensional digital character video with consistent full body movements, facial expressions and voice content semantics; The real-time gesture-driven generation model at least includes an appearance control network, a potential diffusion model and a gesture control module. The process of training the real-time gesture-driven generation model includes: Get the distillation model weights; Initializing a latent diffusion model according to the distillation model weights, and freezing the initialized latent diffusion model; Training the appearance control network using two-dimensional digital human image training data; Training the posture control module with random noise training data to achieve joint fine-tuning of the appearance control network and the posture control network, and returning to the step of training the appearance control network with two-dimensional digital human image training data until the latent diffusion model converges; The converged latent diffusion model is accelerated by quantizing the model to complete the training process of the real-time posture-driven generation model.

2. The method according to claim 1, characterized in that The process of training a full-body motion generation model includes: Acquire training data; the training data at least includes audio training data, whole body movement training data and facial movement expression training data; Extracting Mel spectrum low-level features, high-level features and speech text semantic features from the audio training data; the Mel spectrum low-level features represent the spectrum information of the speech signal; the high-level features represent the semantic information of the speech signal; Inputting the Mel spectrum low-level features, the high-level features and the speech text semantic features into the action speech fusion sharing module to obtain fusion features; Inputting the facial action expression training data, the facial expression random noise and the fusion feature into an action and speech fusion sharing module to obtain a first fusion feature; Inputting the first fused feature into an expression encoder for denoising training to predict facial expression noise training data; Inputting the whole body limb movement training data, limb movement random noise and the fusion feature into the action and speech fusion sharing module to obtain a second fusion feature; Inputting the second fused feature into a posture encoder for denoising training to predict limb noise training data; The whole body motion generation model is trained by using the facial expression noise training data and the body noise training data until the whole body motion generation model reaches convergence.

3. The method according to claim 1, characterized in that The three-dimensional whole-body posture sequence at least includes limb skeleton information and expression coordinate information, and mapping the three-dimensional whole-body posture sequence into a two-dimensional whole-body posture sequence includes: Mapping the limb skeleton information and the expression coordinate information into three-dimensional whole body posture sequence coordinates through a preset mapping method; The three-dimensional whole-body posture sequence coordinates are converted into a two-dimensional whole-body posture sequence.

4. The method according to claim 1, characterized in that: The pre-trained real-time posture drive generation model performs real-time posture drive on the two-dimensional whole body posture sequence and the two-dimensional digital character image to obtain a corresponding posture video, including: Performing posture encoding on the two-dimensional whole-body posture sequence by means of a posture control module in a pre-trained real-time posture-driven generation model; Performing appearance encoding on the two-dimensional digital human image by driving an appearance control network in a real-time posture-generated model; Through the latent diffusion model in the real-time posture-driven generation model and the pre-added random noise, the two-dimensional full-body posture sequence after posture encoding and the two-dimensional digital human image after appearance encoding are diffused and denoised to obtain the corresponding posture video.

5. The method according to claim 1, characterized in that The method of adjusting the facial image of the digital character in the corresponding posture video in real time by using the pre-built lip synchronization model and the audio content so that the lip shape in the facial image is consistent with the audio content to obtain the target digital character video includes: Encode the facial image of the digital person in the corresponding pose video through a variational autoencoder frozen in a pre-built lip sync model; encoding the audio content through a speech recognition model frozen in a lip synchronization model; Through the lip synchronization model in the lip synchronization model, the encoded audio content is used to adjust the encoded facial image in real time, so that the lip shape in the facial image is consistent with the audio content, and the target digital character video is output through the generation network in the lip synchronization model.

6. A real-time digital character generation device, characterized in that: The device comprises: An acquisition unit, used for acquiring audio content and a two-dimensional digital character image; A processing unit, used to process the audio content through a pre-trained whole-body action generation model to obtain a three-dimensional whole-body posture sequence; wherein the whole-body action generation model is obtained by jointly training the facial action expression training data and the whole-body limb action training data guided by the audio training data; A mapping unit, used for mapping the three-dimensional whole-body posture sequence into a two-dimensional whole-body posture sequence; A real-time posture driving unit, used to perform real-time posture driving on the two-dimensional whole-body posture sequence and the two-dimensional digital human image through a pre-trained real-time posture driving generation model to obtain a corresponding posture video; An adjustment unit is used to adjust the facial image of the digital character in the corresponding posture video in real time through a pre-built lip synchronization model and the audio content, so that the lip shape in the facial image is consistent with the audio content, and obtain a target digital character video; wherein the target digital character video is a two-dimensional digital character video with consistent whole body movements, facial expressions and voice content semantics; The real-time posture drive generation model includes an appearance control network, a potential diffusion model and a posture control module, and the real-time posture drive unit for training the real-time posture drive generation model includes: The second acquisition module is used to obtain the distillation model weight; An initialization module, used to initialize the potential diffusion model according to the distillation model weights, and freeze the initialized potential diffusion model; A second training module, used to train the appearance control network using two-dimensional digital character image training data; A third training module is used to train the posture control module through random noise training data to achieve joint fine-tuning of the appearance control network and the posture control network, and return to the step of training the appearance control network through the two-dimensional digital human image training data until the potential diffusion model converges; The acceleration module is used to accelerate the converged potential diffusion model through the quantization model to complete the training process of the real-time posture-driven generation model.

7. The device according to claim 6, characterized in that A processing unit for training a full-body motion generation model, including: A first acquisition module is used to acquire training data; the training data at least includes audio training data, whole body movement training data and facial movement expression training data; An extraction module, used to extract Mel spectrum low-level features, high-level features and speech text semantic features from the audio training data; the Mel spectrum low-level features represent the spectrum information of the speech signal; the high-level features represent the semantic information of the speech signal; A first input module is used to input the Mel spectrum low-level features, the high-level features and the speech text semantic features into the action speech fusion sharing module to obtain fusion features; A second input module is used to input the facial action expression training data, facial expression random noise and the fusion feature into the action and speech fusion sharing module to obtain a first fusion feature; A first denoising training module, used for inputting the first fusion feature into an expression encoder for denoising training to predict facial expression noise training data; A third input module is used to input the whole body limb movement training data, limb movement random noise and the fusion feature into the action and speech fusion sharing module to obtain a second fusion feature; A second denoising training module, used for inputting the second fused features into a posture encoder for denoising training to predict limb noise training data; The first training module is used to train the whole body motion generation model through the facial expression noise training data and the limb noise training data until the whole body motion generation model reaches convergence.

8. The device according to claim 6, characterized in that The three-dimensional whole-body posture sequence includes at least limb skeleton information and expression coordinate information, and the mapping unit includes: A mapping module, used to map the limb skeleton information and the expression coordinate information into three-dimensional whole body posture sequence coordinates through a preset mapping method; The conversion module is used to convert the three-dimensional whole-body posture sequence coordinates into a two-dimensional whole-body posture sequence.

Images