Training method for interactive digital human generation model, generation method, device, storage medium and program product of interactive digital human

Abstract

The present disclosure relates to the field of image generation technology, and in particular to a training method for an interactive digital human generation model, an interactive digital human generation method, an apparatus, a storage medium, and a program product. The interactive digital human generation model includes a large conversation language model and a digital human driving model, and the digital human driving model is connected to a target hidden layer of the large conversation language model. The method includes: inputting a training audio clip into the large conversation language model, outputting a first audio feature vector corresponding to the training audio clip through the target hidden layer; generating a synthetic video frame sequence corresponding to the training audio clip based on the first audio feature vector through the digital human driving model; and updating the parameters of the digital human driving model based on the target video frame sequence and the synthetic video frame sequence corresponding to the training audio clip. The present disclosure can reduce the response delay of an interactive digital human and make the digital human image more natural.

Description

Technical Field

[0001] The present disclosure relates to the field of image generation technology, and in particular to a training method for an interactive digital human generation model, a method for generating an interactive digital human, a training device for an interactive digital human generation model, a device for generating an interactive digital human, a non-volatile computer-readable storage medium, and a computer program product. Background Art

[0002] An interactive digital human is a virtual avatar based on artificial intelligence technology that can interact with users in real time through natural means such as voice, expressions, and movements. These interactive digital humans are typically powered by technologies such as speech recognition, natural language processing, speech synthesis, and computer vision. They are widely used in scenarios such as customer service, education, and live streaming to simulate the experience of real-life interactions. Interactive digital humans can be 2D (two dimensions) or 3D (three dimensions). They analyze user input (such as voice or text) and generate multimodal output (such as voice responses, expressions, and movements), thereby providing more vivid human-computer interaction services. Reducing the latency of interactive digital human generation is a pressing technical challenge. Summary of the Invention

[0003] In view of this, the present disclosure provides a technical solution for generating an interactive digital human.

[0004] According to one aspect of the present disclosure, a method for training an interactive digital human generation model is provided, wherein the interactive digital human generation model includes a large dialogue language model and a digital human driving model, and the digital human driving model is connected to a target hidden layer of the large dialogue language model. The method includes:

[0005] Inputting a training audio segment into the large conversation language model, and outputting a first audio feature vector corresponding to the training audio segment through the target hidden layer;

[0006] generating, by the digital human driving model and based on the first audio feature vector, a synthetic video frame sequence corresponding to the training audio segment;

[0007] The parameters of the digital human driving model are updated according to the target video frame sequence corresponding to the training audio segment and the synthesized video frame sequence.

[0008] In a possible implementation, the target hidden layer is the last hidden layer of the large dialogue language model.

[0009] In a possible implementation, the digital human driving model includes an audio encoder, an image encoder, and an image decoder, and the audio encoder in the digital human driving model is connected to the target hidden layer;

[0010] Generating a synthetic video frame sequence corresponding to the training audio segment based on the first audio feature vector by using the digital human driving model includes:

[0011] Encoding the first audio feature vector by an audio encoder in the digital human driving model to obtain a second audio feature vector;

[0012] Encoding the reference video frame by the image encoder to obtain a reference image feature vector;

[0013] The second audio feature vector and the reference image feature vector are decoded by the image decoder to obtain a synthetic video frame sequence corresponding to the training audio segment.

[0014] In a possible implementation, during the parameter update process of the digital human driving model, the parameters of the large dialogue language model remain fixed.

[0015] In a possible implementation, updating the parameters of the digital human driving model according to the target video frame sequence and the synthesized video frame sequence corresponding to the training audio segment includes:

[0016] Calculating pixel-level loss between the target video frame sequence and the synthesized video frame sequence;

[0017] Based on the pixel-level loss, the parameters of the digital human driving model are updated through back propagation.

[0018] In a possible implementation, before inputting the training audio segment into the large dialogue language model, the method further includes:

[0019] Performing speech recognition training and speech synthesis training on the large conversation language model using audio-text pairs;

[0020] The large conversational language model is trained for question answering using text-to-text pairs.

[0021] In one possible implementation, the large conversational language model alternately outputs text tags and audio tags during training.

[0022] According to another aspect of the present disclosure, a method for generating an interactive digital human is provided, comprising:

[0023] Get the audio to be replied;

[0024] Input the audio to be replied to into the interactive digital human generation model, and output the reply text, reply audio and digital human video corresponding to the audio to be replied to through the interactive digital human generation model;

[0025] The interactive digital human generation model is trained using the training method for the interactive digital human generation model.

[0026] According to another aspect of the present disclosure, a training device for an interactive digital human generation model is provided, wherein the interactive digital human generation model includes a large dialogue language model and a digital human driving model, and the digital human driving model is connected to a target hidden layer of the large dialogue language model. The device includes:

[0027] an extraction module, configured to input a training audio segment into the large dialogue language model and output a first audio feature vector corresponding to the training audio segment through the target hidden layer;

[0028] A generation module, configured to generate a synthetic video frame sequence corresponding to the training audio segment based on the first audio feature vector by using the digital human driving model;

[0029] An updating module is used to update the parameters of the digital human driving model according to the target video frame sequence corresponding to the training audio segment and the synthesized video frame sequence.

[0030] In a possible implementation, the target hidden layer is the last hidden layer of the large dialogue language model.

[0031] In a possible implementation, the digital human driving model includes an audio encoder, an image encoder, and an image decoder, and the audio encoder in the digital human driving model is connected to the target hidden layer;

[0032] The generation module is used to:

[0033] Encoding the first audio feature vector by an audio encoder in the digital human driving model to obtain a second audio feature vector;

[0034] Encoding the reference video frame by the image encoder to obtain a reference image feature vector;

[0035] The second audio feature vector and the reference image feature vector are decoded by the image decoder to obtain a synthetic video frame sequence corresponding to the training audio segment.

[0036] In a possible implementation, during the parameter update process of the digital human driving model, the parameters of the large dialogue language model remain fixed.

[0037] In a possible implementation, the update module is configured to:

[0038] Calculating pixel-level loss between the target video frame sequence and the synthesized video frame sequence;

[0039] Based on the pixel-level loss, the parameters of the digital human driving model are updated through back propagation.

[0040] In a possible implementation, the apparatus further includes:

[0041] A first training module is used to perform speech recognition training and speech synthesis training on the large dialogue language model using audio-text pairs;

[0042] The second training module is used to perform question-answering training on the dialogue language model using text-text pairs.

[0043] In one possible implementation, the large conversational language model alternately outputs text tags and audio tags during training.

[0044] According to another aspect of the present disclosure, there is provided a device for generating an interactive digital human, comprising:

[0045] The acquisition module is used to obtain the audio to be replied;

[0046] An output module, configured to input the audio to be replied to into the interactive digital human generation model, and output the reply text, reply audio and digital human video corresponding to the audio to be replied to via the interactive digital human generation model;

[0047] The interactive digital human generation model is obtained by training using the interactive digital human generation model training device.

[0048] According to another aspect of the present disclosure, a training device for generating an interactive digital human model is provided, comprising a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above method.

[0049] According to another aspect of the present disclosure, a device for generating an interactive digital human is provided, comprising a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above method.

[0050] According to another aspect of the present disclosure, a non-volatile computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above method are implemented.

[0051] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, or a non-volatile computer-readable storage medium carrying the computer program, wherein the computer program implements the steps of the above method when executed by a processor.

[0052] In an embodiment of the present disclosure, the interactive digital human generation model includes a large conversational language model and a digital human driving model, and the digital human driving model is connected to the target hidden layer of the large conversational language model. A training audio clip is input into the large conversational language model, and the target hidden layer outputs a first audio feature vector corresponding to the training audio clip. The digital human driving model generates a synthetic video frame sequence corresponding to the training audio clip based on the first audio feature vector. The parameters of the digital human driving model are then updated based on the target video frame sequence and the synthetic video frame sequence corresponding to the training audio clip. This allows the interactive digital human generation model to integrate the large conversational language model and the digital human driving model end-to-end. A single interactive digital human generation model can simultaneously generate output data in three modalities: text response, audio response, and digital human image, based on the user's input speech, avoiding the serial delay of traditional cascaded systems. The digital human driving model directly utilizes the hidden layer features of the large conversational language model, avoiding the delay of waiting for complete audio synthesis. This reduces the response delay of the interactive digital human and improves the user experience. Compared to interactive digital human systems in related technologies that include multiple modules, the interactive digital human generation model provided by the disclosed embodiments integrates the functions of multiple modules, making the model smaller and easier to deploy. Furthermore, the disclosed embodiments directly use the output of the hidden layer of the large conversational language model, rather than synthesized speech, as the digital human drive signal. This allows the generated digital human image to better match the paralinguistic information (such as tone, emotion, and background noise) of the audio response, making the digital human image more natural.

[0053] Further features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate exemplary embodiments, features, and aspects of the disclosure and, together with the description, serve to explain the principles of the disclosure.

[0055] Figure 1 A schematic diagram showing a classic interactive digital human system in related technologies.

[0056] Figure 2 Schematic diagram showing an autoregressive large language model.

[0057] Figure 3 A schematic diagram showing a speech recognition model based on an autoregressive large language model.

[0058] Figure 4 A schematic diagram showing a speech synthesis model based on an autoregressive large language model.

[0059] Figure 5A schematic diagram showing the integration of speech recognition, a large language model, and speech synthesis into a unified end-to-end dialogue model.

[0060] Figure 6 A schematic diagram showing a large model of an end-to-end conversation.

[0061] Figure 7a Schematic diagram showing a classic interactive digital human system and its response delay.

[0062] Figure 7b A schematic diagram showing an interactive digital human system based on a large dialogue language model and a digital human driving model and its response delay.

[0063] Figure 7c A schematic diagram showing an interactive digital human system and its response delay provided by an embodiment of the present disclosure.

[0064] Figure 8 A flowchart of a method for training an interactive digital human generation model provided by an embodiment of the present disclosure is shown.

[0065] Figure 9 A schematic diagram showing a method for training an interactive digital human generation model provided by an embodiment of the present disclosure, in which a large dialogue language model alternately outputs N text tags and M audio tags during training.

[0066] Figure 10 Schematic diagram showing the original Wav2lip model structure and workflow.

[0067] Figure 11 A schematic diagram of the workflow of an interactive digital human system based on a large dialogue language model and a Wav2lip model in related technologies is shown.

[0068] Figure 12 A schematic diagram of an interactive digital human generation model provided by an embodiment of the present disclosure is shown.

[0069] Figure 13 Another schematic diagram of the interactive digital human generation model provided by an embodiment of the present disclosure is shown.

[0070] Figure 14 A block diagram of a training device for generating an interactive digital human model provided by an embodiment of the present disclosure is shown.

[0071] Figure 15 It is a block diagram of a training device for generating an interactive digital human model or a generating device 1900 for an interactive digital human according to an exemplary embodiment. DETAILED DESCRIPTION

[0072] Various exemplary embodiments, features, and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise indicated.

[0073] As used herein, the terms "comprises," "comprising," "having," or variations thereof are open ended and include one or more stated features, integers, elements, steps, parts, or functions, but do not preclude the presence or addition of one or more other features, integers, elements, steps, parts, functions, or groups thereof.

[0074] When an element is referred to as being "connected," "coupled," "responsive" or variations thereof to another element, it can be directly connected, coupled or responsive to the other element or intervening elements may be present.

[0075] Although the terms first, second, third, etc. may be used herein to describe various elements / operations, these elements / operations should not be limited by these terms. These terms are only used to distinguish one element / operation from another element / operation. Therefore, without departing from the teachings of the present invention, the first element / operation in some embodiments may be referred to as the second element / operation in other embodiments.

[0076] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0077] In addition, numerous specific details are provided in the following detailed description to better illustrate the present disclosure. Those skilled in the art will appreciate that the present disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art are not described in detail in order to highlight the main points of the present disclosure.

[0078] With the development of large language model technology, interactive digital humans are being gradually applied to a growing number of human-computer interaction scenarios. On live streaming and short video platforms, the use of virtual anchors can save labor costs and enable 24 / 7 uninterrupted operation. In educational scenarios, the use of digital teachers can attract students' attention and improve learning outcomes. In customer service scenarios, digital customer service humans are more approachable than simple text or voice customer service.

[0079] Figure 1 A schematic diagram of a classic interactive digital human system in related technology is shown. Figure 1In the example shown, the interactive digital human system consists of several modules, including speech recognition, a large language model (LLM), speech synthesis, and a digital human driver. In a classic interactive digital human system, to achieve a realistic interactive digital human, after the system hears the user's voice, it undergoes speech recognition, large language model analysis, speech synthesis, and digital human driver. These components operate sequentially: when the user generates voice input, speech recognition transcribes the speech into text; the large language model generates a text response based on the input text; speech synthesis converts the text into audio; and the digital human driver model generates a digital human image based on the audio and displays it on a display device.

[0080] Among these steps, the accuracy of speech recognition determines whether the system can understand what the user is saying; the feedback from the large language model determines the system's intelligence; and the effectiveness of speech synthesis and digital human-driven processing determines the degree of human-likeness of the digital human. In addition to these factors, the latency between the interactive digital human hearing the user's question and the digital human's response—that is, the digital human's response speed—can significantly impact the user experience. Because these components operate serially, the overall system latency is the sum of the delays of each component. Due to the long link, the overall latency is high, significantly impacting the user experience. Although each component may perform well in recognition or synthesis individually, it is difficult to reduce the overall latency. Furthermore, because speech recognition converts audio into text, the large language model cannot respond based on the speech's paralinguistic information (such as tone, emotion, and background noise).

[0081] At present, in addition to text processing, methods based on large language models are gradually being applied to speech recognition and speech synthesis.

[0082] In speech recognition, using large language models can achieve better recognition results. For example, methods such as SLAM-LLM (Speech-Language Model Augmented with Large Language Model) and MooER transform large language models from traditional text input to speech input, leveraging the vast knowledge stored in large language models to reduce recognition error rates.

[0083] In speech synthesis, the use of large language models based on autoregressive (AR) technology has also achieved excellent results. For example, methods such as VALL-E (Neural Codec Language Model for Text-to-Speech Synthesis), BASE-TTS (Big Adaptive Streamable TTS with Emergent Abilities), and Seed-TTS (Scalable Efficient Emotional Diffusion-based TTS) can synthesize audio that closely resembles the intonation and rhythm of human speech.

[0084] Figure 2 A schematic diagram of an autoregressive large language model is shown. Figure 2 As shown, the autoregressive large language model can process the text tokens (corresponding to Figure 2 ) and generates the corresponding reply token (corresponding to Figure 2 In interactive digital human applications, supervised fine-tuning (SFT) or low-rank adaptation (LoRA) of large language models is often performed using domain-specific data to improve their response capabilities in specialized fields. At the same time, database knowledge can be imported using retrieval-augmented generation (RAG) technology to significantly improve the quality and professionalism of the large language model's responses.

[0085] Figure 3 Schematic diagram of a speech recognition model based on an autoregressive large language model is shown. Figure 3 As shown in the figure, user audio (i.e., user input) is processed by the audio encoder to generate an audio feature vector (embedding). To adapt to the existing large language model, the audio feature vector is mapped from the audio feature space to the text feature space through a modal conversion model (i.e., an adapter). Subsequently, the corresponding text tokens are recognized based on the fine-tuned large language model, completing the speech recognition process.

[0086] When training a speech recognition model based on a large language model, you can use the training audio as the model input, keep the audio encoder parameters fixed, and focus on training the modal conversion model. At the same time, you can use supervised fine-tuning or low-rank adaptation techniques to optimize the speech recognition model based on the large language model so that its output text tags are as consistent as possible with the annotated text of the training audio.

[0087] Figure 4 The figure shows a schematic diagram of a speech synthesis model based on an autoregressive large language model. The speech synthesis model can perform discrete coding on the audio signal. Discrete coding can be achieved through methods such as Vector Quantized Variational Autoencoder (VQ-VAE) to achieve unsupervised coding, or supervised coding can be achieved by embedding a Vector Quantization (VQ) layer in the speech recognition model. The input of the speech synthesis model can include the text to be synthesized and the reference audio for controlling the synthesized timbre. The voiceprint encoder can extract the voiceprint vector corresponding to the reference audio. The output of the large language model in the speech synthesis model can be a discrete token sequence (corresponding to Figure 4 ). Audio decoders such as HiFi-GAN (High-Fidelity Generative Adversarial Network) can be used to decode the discrete token sequence to obtain the target audio.

[0088] like Figures 2 to 4 As shown, autoregressive large language model technology can be used in speech recognition models, large language models, and speech synthesis models. Technologies such as GLM-4-Voice, Moshi, and GPT-4 integrate these components into a single large language model, an end-to-end conversational large language model, to process input speech and directly generate text and audio responses.

[0089] Figure 5 The following diagram shows how speech recognition, large language model, and speech synthesis are integrated into a unified end-to-end dialogue model. Figure 5 As shown, in the end-to-end dialogue language model, the user input speech (corresponding to Figure 5 The "input audio" in the source file is processed by the end-to-end conversational language model to directly generate reply text and reply audio.

[0090] Figure 6 A schematic diagram of the end-to-end dialogue model is shown. Figure 6As shown, the end-to-end conversational large model integrates the speech recognition model, the large language model, and the speech synthesis model based on the large language model into a unified model. From a model perspective, the end-to-end conversational large language model integrates the various modules of the speech recognition model, the large language model, and the speech synthesis model based on the large language model, with the large language model at its core. The input to the autoregressive large language model in the end-to-end conversational large language model can include the audio feature vector obtained by passing the user audio through the audio encoder and modality converter, and the voiceprint vector obtained by passing the reference audio through the voiceprint encoder. The voiceprint vector can be used to control the timbre of the synthesized audio. After the voiceprint vector and the audio feature vector corresponding to the user audio are processed by the large language model, text tags and audio tags are output in sequence. The audio tags can be converted into synthesized audio after passing through the audio decoder.

[0091] End-to-end conversational language models can reduce latency to a certain extent and understand the paralinguistic information in user speech. However, in interactive digital human systems, the conversational language model still needs to generate a certain length of audio before it can start driving the digital human model and generating a displayable digital human image. This cascaded operation still results in a relatively high overall latency.

[0092] To address technical issues similar to those described above, embodiments of the present disclosure provide a training method for an interactive digital human generation model. The interactive digital human generation model includes a large conversational language model and a digital human driving model, wherein the digital human driving model is connected to a target hidden layer of the large conversational language model. A training audio clip is input into the large conversational language model, and the target hidden layer outputs a first audio feature vector corresponding to the training audio clip. The digital human driving model generates a synthetic video frame sequence corresponding to the training audio clip based on the first audio feature vector. The parameters of the digital human driving model are then updated based on the target video frame sequence and the synthetic video frame sequence corresponding to the training audio clip. This method integrates the large conversational language model and the digital human driving model through an end-to-end interactive digital human generation model. A single interactive digital human generation model can simultaneously generate output data in three modalities: text response, audio response, and digital human image, based on the user's input speech, thereby avoiding the serial delay of traditional cascaded systems. The digital human driving model directly utilizes the hidden layer features of the large conversational language model, avoiding the delay of waiting for complete audio synthesis. This reduces the response delay of the interactive digital human and improves the user experience. Compared to interactive digital human systems in related technologies that include multiple modules, the interactive digital human generation model provided by the disclosed embodiments integrates the functions of multiple modules, making the model smaller and easier to deploy. Furthermore, the disclosed embodiments directly use the output of the hidden layer of the large conversational language model, rather than synthesized speech, as the digital human drive signal. This allows the generated digital human image to better match the paralinguistic information (such as tone, emotion, and background noise) of the audio response, making the digital human image more natural.

[0093] Figure 7a A schematic diagram showing a classic interactive digital human system and its response delay. Figure 7a As shown, in a classic interactive digital human system, when a user generates voice input, speech recognition transcribes the speech into text (for example, the speech recognition result is "Hello Xiaomo, may I ask..."). A large language model generates a text response based on the speech recognition result (for example, the large language model's response is "Hello, welcome to use..."). Speech synthesis generates synthesized audio based on the response text generated by the large language model. The digital human driving model generates a digital human image based on the synthesized audio. In classic interactive digital human systems, response delays are significant.

[0094] Figure 7b A schematic diagram shows an interactive digital human system based on a large conversational language model and a digital human driver model, along with its response delay. In this type of interactive digital human system, speech recognition, a large language model, and speech synthesis are integrated to create an end-to-end large conversational language model. The end-to-end large conversational language model generates text and audio responses, while the digital human driver model generates the digital human image. Figure 7a and Figure 7b It can be seen that this type of interactive digital human system can reduce response delay compared with the classic interactive digital human system.

[0095] Figure 7c A schematic diagram illustrating the interactive digital human system and its response delay provided by an embodiment of the present disclosure. In this embodiment, the large conversational language model and the digital human driving model are integrated into an end-to-end interactive digital human generation model. This single interactive digital human generation model generates output data in three modalities: text responses, audio responses, and the digital human's image. This can further reduce the interactive digital human's response delay.

[0096] In addition, the disclosed embodiment can be modified based on any existing large dialogue language model, and the digital human-driven model can be integrated into the existing large dialogue language model. There is no need to retrain the large language model. In the post-training stage, only the parameters of the digital human-driven model need to be adjusted, saving a lot of training costs.

[0097] The following describes in detail the training method for generating an interactive digital human model provided by the embodiments of the present disclosure in conjunction with the accompanying drawings.

[0098] Figure 8 A flow chart of the training method for an interactive digital human generation model provided by an embodiment of the present disclosure is shown. In one possible implementation, the execution subject of the training method for an interactive digital human generation model may be a training device for an interactive digital human generation model. For example, the training method for an interactive digital human generation model may be executed by a terminal device or a server or other electronic device. The terminal device may be a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device or a wearable device, etc. In some possible implementations, the training method for an interactive digital human generation model may be implemented by a processor calling computer-readable instructions stored in a memory. In an embodiment of the present disclosure, the interactive digital human generation model includes a large dialogue language model and a digital human driving model, and the digital human driving model is connected to the target hidden layer of the large dialogue language model. As Figure 8 As shown, the training method for generating an interactive digital human model includes steps S11 to S13.

[0099] In step S11, a training audio segment is input into the dialogue language model, and a first audio feature vector corresponding to the training audio segment is output through the target hidden layer.

[0100] In step S12, a synthetic video frame sequence corresponding to the training audio segment is generated based on the first audio feature vector by the digital human driving model.

[0101] In step S13, the parameters of the digital human driving model are updated according to the target video frame sequence corresponding to the training audio segment and the synthesized video frame sequence.

[0102] In the disclosed embodiment, the interactive digital human generation model is an end-to-end system that integrates a large dialogue language model and a digital human driving model, which can achieve synchronous generation of multimodal outputs through the connection of target hidden layers.

[0103] In one possible implementation, the large conversational language model may be a Transformer-based autoregressive model.

[0104] As an example of this implementation, the MooER large model can be used as the large conversational language model. Of course, those skilled in the art can flexibly select a large conversational language model based on actual application scenario requirements and / or personal preferences, and this is not limited here.

[0105] In the disclosed embodiments, the conversational language model can process speech input and generate text and audio responses. The output layer of the conversational language model can include two branches: text and audio.

[0106] In one possible implementation, before inputting the training audio clip into the large dialogue language model, the method further includes: using audio-text pairs to perform speech recognition training and speech synthesis training on the large dialogue language model; and using text-text pairs to perform question-answering training on the large dialogue language model.

[0107] As an example of this implementation, a large conversational language model can be trained using audio-text pairs, enabling it to convert input speech into text (speech recognition capability). In this example, supervised training of the large conversational language model can be performed using a large amount of audio-text pair data (such as user speech clips and their corresponding transcripts). For example, the input audio can be converted into a feature vector using an audio encoder (such as a convolutional neural network or Transformer), and then mapped to a text feature space using a modal converter (Adapter). The large conversational language model can be fine-tuned so that its output text tags are consistent with the annotated text. In one example, when using audio-text pairs to train the speech recognition capability of the large conversational language model, the parameters of the audio encoder of the large conversational language model can be fixed, and only the modal converter and the adaptation layer of the large conversational language model (such as LoRA) can be trained. In addition, a cross-entropy loss of the text tags can be used.

[0108] As an example of this implementation, a large conversational language model can be trained using audio-text pairs, enabling it to synthesize text responses into speech (speech synthesis capability). In this example, the same audio-text pair data as the previous example can be used, but the training objective is to generate audio tokens (such as audio tokens discretely encoded by a VQ-VAE). During training, text tokens and voiceprint features of reference audio can be input. The large conversational language model outputs the corresponding audio token sequence, which is then restored to speech through a decoder (such as HiFi-GAN). In one example, audio tokens can be generated using an autoregressive method and output alternately with text tokens (e.g., outputting N text tokens first, followed by M audio tokens).

[0109] As an example of this implementation, question-answering training can be performed on a large conversational language model using text-to-text pairs to improve its semantic understanding and text generation capabilities (conversational IQ). In this example, supervised fine-tuning (SFT) can be performed using text-to-text data (such as question-answering datasets and conversation logs), and external knowledge bases can be introduced through retrieval-augmented generation (RAG). In one example, techniques such as low-rank adaptation (LoRA) can be used for efficient fine-tuning while preserving pre-trained knowledge.

[0110] This implementation method uses phased training (speech recognition, speech synthesis, and question-answering training) to enable the large dialogue language model to have multimodal processing capabilities. It can directly understand speech input and generate text and voice responses, while combining question-answering training to improve the accuracy of semantic understanding.

[0111] In another possible implementation, an existing large conversation language model can be directly obtained without training the large conversation language model.

[0112] In embodiments of the present disclosure, the output of the large conversational language model may include text tags and audio tags. In one possible implementation, the output layer of the large conversational language model may include two independent submodules: a text output head and an audio output head. The text output head may be responsible for generating text tags (e.g., outputting a text vocabulary distribution via a softmax function). The audio output head may be responsible for generating audio tags (e.g., discrete audio tags encoded via a VQ-VAE).

[0113] In one possible implementation, the large conversational language model alternately outputs text tags and audio tags during training. Figure 9 A schematic diagram showing a method for training an interactive digital human generation model provided by an embodiment of the present disclosure, in which a large dialogue language model alternately outputs N text tags and M audio tags during training.

[0114] In this implementation, training data can be constructed by structuring conversation samples into alternating sequences, for example: [text token_1...N][audio token_1...M][text token_N+1...][audio token_M+1...]. Audio segments can be discretized into token sequences using an encoder such as VQ-VAE. Text tokens and audio tokens are alternating in a strictly predetermined ratio (e.g., N:M) in the training data.

[0115] An autoregressive training objective can be used. The large conversational language model needs to predict the type (text / audio) and content of the next token. A loss function can be used to force the model to learn alternating patterns: text cross-entropy loss + audio token contrast loss + modality switching penalty. For example, after generating N consecutive text tokens, incorrectly predicting a text token will result in a higher loss.

[0116] After being trained with massive alternating samples, the large conversational language model can implicitly establish modal switching rules internally, thereby automatically maintaining the alternation ratio in the training data during inference.

[0117] This implementation, through training data design, enables the large conversational language model to alternately output text and audio tokens, enabling multimodal streaming generation without relying on an external controller. This design enables the large conversational language model to generate text responses and corresponding audio segments simultaneously in real time, eliminating the latency of traditional cascaded systems that must first generate complete text and then synthesize speech, significantly reducing interaction latency.

[0118] In one possible implementation, Wav2lip can be used as the digital human driving model. Of course, those skilled in the art can flexibly select a digital human driving model based on actual application scenario requirements and / or personal preferences, and this is not a limitation here. For example, the digital human driving model can also adopt a neural voice puppetry model.

[0119] Figure 10 The schematic diagram of the original Wav2lip model structure and workflow is shown in Figure 2. Figure 10 As shown in Figure 2, the Wav2lip model synthesizes digital human videos using frame sequence processing. The Wav2lip model employs a two-way parallel processing architecture: one input takes a continuous multi-frame reference video (e.g., frames N-2, N-1, N, N+1, and N+2), extracting visual feature vectors through an image encoder; the other inputs segmented audio data, extracting acoustic feature vectors through an audio encoder. The Wav2lip model then feeds the visual and audio feature vectors into an image decoder for fusion processing, ultimately generating the Nth frame of synthesized video output, synchronizing the digital human's lip movements with the input audio.

[0120] Interactive digital humans can be categorized into two main categories: 2D (two-dimensional) and 3D (three-dimensional). 3D digital humans are created by artists using software like Unreal Engine (UE), with corresponding expressions and movements generated using motion capture and other technologies. Compared to 3D digital humans, the production process for 2D digital humans is simpler, less expensive, and more customizable. Currently, technologies like Wav2lip and neural speech drive only require a few minutes of spoken video of the target digital human to generate a corresponding 2D digital human model, which is then used to drive the target digital human image. This significantly simplifies the digital human production process and expands the scope of digital human applications.

[0121] Figure 11 The following is a schematic diagram showing the workflow of an interactive digital human system based on a large dialogue language model and a Wav2lip model in the related art. Figure 11 As shown in the figure, in related art, the audio tokens generated by the large dialogue language model are converted into the final output audio signal through an audio decoder. The Wav2lip model must wait for the audio decoder to complete the synthesis of a complete audio segment (chunk) before it can drive the generation of the digital human video based on this audio. In related art, because audio synthesis and video generation are performed sequentially, video generation must wait until the audio is fully synthesized before starting, resulting in significant interaction delays.

[0122] In the embodiment of the present disclosure, the digital human driving model may be responsible for generating a digital human video frame sequence synchronized with audio. Figure 12 A schematic diagram of an interactive digital human generation model provided by an embodiment of the present disclosure is shown. Figure 12 As shown, in the interactive digital human generation model provided by the embodiments of the present disclosure, the input to the digital human driving model comes directly from the output of the target hidden layer of the large conversational language model, rather than the synthesized audio in traditional cascade systems. That is, after processing the user audio input, the large conversational language model not only outputs text and audio tags, but also the audio feature vector (i.e., the first audio feature vector) generated by its target hidden layer is directly used as the input to the digital human driving model. In other words, in the embodiments of the present disclosure, the input audio of the digital human driving model in traditional interactive digital human systems is replaced by the output of the target hidden layer of the large conversational language model.

[0123] like Figure 12As shown, the interactive digital human generation model receives inputs including reference audio and user audio. The conversational language model processes the user audio using an audio encoder and a text-to-speech modality converter, while the reference audio is processed through a voiceprint encoder to generate a voiceprint vector. The conversational language model generates text and audio tags. The audio tags are fed into an audio decoder to synthesize the final audio output.

[0124] At the same time, the first audio feature vector generated by the target hidden layer of the large conversational language model can be directly passed to the digital human driving model, rather than waiting for the audio decoder to complete audio synthesis. The digital human driving model uses the first audio feature vector and reference facial information (such as a reference facial image or video) to generate a sequence of digital human video frames synchronized with the audio. This disclosed embodiment enables the digital human driving model to process in parallel with the audio decoder in the large conversational language model, significantly reducing interaction latency and achieving faster digital human response.

[0125] In a possible implementation, the target hidden layer is the last hidden layer of the large dialogue language model.

[0126] In a neural network, a hidden layer is a layer located between the input and output layers. It is responsible for extracting features from the input data and creating internal representations. Deep learning models, especially large language models, typically have multiple hidden layers, each attempting to capture different levels of features and abstractions in the input data.

[0127] In this implementation, the digital human-driven model can directly use the output of the last hidden layer in the large conversational language model as its input. The last hidden layer refers to the layer closest to the output layer among the multiple hidden layers in the large conversational language model.

[0128] In this implementation, the last hidden layer of the large conversational language model can capture and integrate rich semantic information and contextual features from all previous layers, thereby providing a more accurate and comprehensive audio feature vector for the digital human-driven model. This audio feature vector not only contains the semantic content of the text, but also contains paralinguistic information such as the rhythm, emotion, and style of the speech. Therefore, using this audio feature vector to drive the digital human-driven model can generate digital human videos that are more natural, realistic, and highly synchronized with the audio content, thereby significantly improving the expressiveness of the interactive digital human and the user's interactive experience. At the same time, this implementation also helps to reduce the computational complexity and inference latency of the model because it avoids additional processing of the output of early layers, making the overall system more efficient.

[0129] Of course, in other possible implementations, the target hidden layer may also be another hidden layer of the large dialogue language model. For example, the target hidden layer may also be the penultimate hidden layer of the large dialogue language model, and so on.

[0130] In an embodiment of the present disclosure, the digital human driving model can be trained based on an audio-visual synchronization training data set. The audio-visual synchronization training data set may include multiple audio-visual synchronization training data, and any audio-visual synchronization training data may include a pair of aligned training audio clips and target video frame sequences. The training audio clip may be a short audio segment (such as a 0.5-second speech waveform or Mel spectrum). The target video frame sequence may be a speaking video frame of the target digital human (such as 5 consecutive frames of facial images) synchronized with the training audio clip. For example, if the audio sampling rate is 16kHz and the video is 25FPS, then 1 second of audio (16,000 sample points) needs to correspond to 25 frames of video, and each frame is aligned with 640 audio sample points.

[0131] In some application scenarios, the target video frame sequence may also be referred to as a real video frame sequence, a real audio-visual synchronized video frame sequence, a ground truth video frame sequence, etc., which is not limited here.

[0132] In an embodiment of the present disclosure, for any item of audio-visual synchronization training data, a training audio segment in the audio-visual synchronization training data can be input into a large dialogue language model. The large dialogue language model internally contains multiple layers, in which the hidden layer is responsible for extracting features from the input data. In the multi-layer structure of the large dialogue language model, the target hidden layer (which may be the last layer or other specified layer) can be used to output features related to the input audio segment. After the target hidden layer processes the training audio segment, it generates a first audio feature vector. The first audio feature vector is a high-level representation of the training audio segment and contains key information in the training audio segment. The first audio feature vector is then used as input to the digital human-driven model to generate a synthetic video frame sequence.

[0133] In one possible implementation, the digital human-driven model includes an audio encoder, an image encoder, and an image decoder, and the audio encoder in the digital human-driven model is connected to the target hidden layer; generating a synthetic video frame sequence corresponding to the training audio segment based on the first audio feature vector by the digital human-driven model includes: encoding the first audio feature vector by the audio encoder in the digital human-driven model to obtain a second audio feature vector; encoding a reference video frame by the image encoder to obtain a reference image feature vector; and decoding the second audio feature vector and the reference image feature vector by the image decoder to obtain a synthetic video frame sequence corresponding to the training audio segment.

[0134] In this implementation, the digital human driving model includes an audio encoder, an image encoder, and an image decoder. The audio encoder processes audio information, the image encoder processes image information, and the image decoder generates the final video frame based on the encoded audio and image features.

[0135] In this implementation, the audio encoder in the digital human-driven model is connected to the target hidden layer of the large dialogue language model to obtain the first audio feature vector output by the target hidden layer. The first audio feature vector can contain paralinguistic information such as the semantics, tone, and emotion of the speech.

[0136] In this implementation, after the large conversational language model outputs the first audio feature vector, the digital human-driven model and the audio decoder in the large conversational language model can immediately begin processing simultaneously: the digital human-driven model generates a synthetic video frame sequence (digital human video) based on the first audio feature vector, while the audio decoder simultaneously converts the first audio feature vector into synthetic audio. This parallel processing mechanism eliminates the serial delay in traditional systems, where audio synthesis must complete before digital human generation can begin, significantly reducing overall interaction latency.

[0137] In this implementation, the audio encoder in the digital human-driven model further encodes the first audio feature vector to generate a second audio feature vector. The image encoder processes a reference video frame (e.g., a neutral expression image of the target digital human or a continuous multi-frame video clip) to generate a reference image feature vector. The reference image feature vector captures the visual features of the target digital human, such as facial expression and head pose. The image decoder combines the second audio feature vector with the reference image feature vector to decode and generate a synthetic video frame sequence corresponding to the training audio clip. This process uses audio features to guide video frame generation, ensuring that the digital human's lip movements, expressions, and movements are synchronized with the input audio.

[0138] As an example of this implementation, the audio encoder can be a convolutional neural network or a Transformer architecture.

[0139] As an example of this implementation, the image decoder may be a Generative Adversarial Network (GAN) or a diffusion model.

[0140] In this implementation, the digital human driving model can quickly generate a sequence of digital human video frames synchronized with the audio based on the audio feature vectors output by the large dialogue language model. This implementation not only improves the efficiency of digital human generation, but also ensures that the digital human's performance is more natural and realistic, thereby enhancing the user interaction experience.

[0141] In an embodiment of the present disclosure, the parameters of the digital human driving model can be updated based on the target video frame sequence corresponding to the training audio clip and the synthesized video frame sequence. The target video frame sequence is a series of ideal video frames that serve as a reference standard for matching the content of the training audio clip. The target video frame sequence demonstrates the expressions and movements that a digital human should have when hearing specific audio. In other words, the target video frame sequence represents the desired movements and expressions of a digital human. The synthesized video frame sequence is a video frame sequence generated by the digital human driving model based on an input audio feature vector (e.g., a first audio feature vector). The digital human driving model attempts to generate video frames similar to the target video frame sequence.

[0142] During training, the parameters of the digital human-driven model (e.g., weights and biases in the neural network) can be adjusted based on the differences between the synthesized video frame sequence and the target video frame sequence. This can be achieved, for example, by calculating a loss function between the synthesized and target video frame sequences, which measures the inconsistency between the synthesized and target video frame sequences. The value of the loss function can be used to update the parameters of the digital human-driven model via a backpropagation algorithm. Backpropagation is an optimization algorithm that adjusts the parameters of the digital human-driven model based on the gradient of the loss function with respect to these parameters. The goal is to minimize the loss function, thereby bringing the synthesized video frame sequence closer to the target video frame sequence.

[0143] In one possible implementation, updating the parameters of the digital human driving model based on the target video frame sequence and the synthetic video frame sequence corresponding to the training audio clip includes: calculating the pixel-level loss between the target video frame sequence and the synthetic video frame sequence; and updating the parameters of the digital human driving model through back propagation based on the pixel-level loss.

[0144] In this implementation, to measure the similarity between the synthesized video frame sequence and the target video frame sequence, a pixel-level loss can be calculated. Specifically, the color difference between each pixel in the synthesized and target video frame sequences can be calculated. The loss function can be the mean squared error (MSE), the structural similarity index (SSIM), or other similar metrics. After calculating the pixel-level loss, the pixel-level loss value can be used to guide parameter updates in the digital human driving model.

[0145] This implementation optimizes the digital human driving model by directly comparing the pixel-level differences between the target video frame sequence and the synthesized video frame sequence, ensuring that the generated digital human video is visually highly consistent with the intended movements and expressions. This implementation utilizes a backpropagation algorithm to precisely adjust the parameters of the digital human driving model to minimize the differences between the target and synthesized video frame sequences, improving the accuracy and quality of video frame synthesis. Through continuous iteration and parameter updates, the digital human driving model can learn to generate more realistic and synchronized digital human videos, significantly improving the end-user's interactive experience and making the digital human's visual performance more natural and expressive.

[0146] Of course, those skilled in the art can flexibly select the type of loss function based on actual application requirements and / or personal preferences. For example, in addition to pixel-level loss, feature-based loss (such as perceptual loss) can also be used to optimize by comparing the similarity between the target video frame sequence and the synthesized video frame sequence in the deep feature space.

[0147] In a possible implementation, during the parameter update process of the digital human driving model, the parameters of the large dialogue language model remain fixed.

[0148] In this implementation, after the training of the large dialogue language model is completed, the parameters of the large dialogue language model are kept fixed, and only the parameters of the digital human driving model are updated.

[0149] This implementation significantly improves the development and deployment efficiency of interactive digital human generation systems. By keeping the parameters of the large conversational language model fixed, its already acquired rich language knowledge and processing capabilities can be leveraged, avoiding the high computational cost of repeatedly training this complex model. Furthermore, by updating only the parameters of the digital human-driven model, the system can quickly adapt to specific application requirements and optimize the quality of digital human video generation. This implementation not only saves time and resources but also accelerates model iteration, enabling the system to more flexibly respond to product development and market changes, ultimately improving user experience and system practicality.

[0150] The disclosed embodiment also provides a method for generating an interactive digital human, comprising: obtaining audio to be replied; inputting the audio to be replied into an interactive digital human generation model, and outputting reply text, reply audio and digital human video corresponding to the audio to be replied through the interactive digital human generation model; wherein the interactive digital human generation model is trained using the training method of the interactive digital human generation model.

[0151] In the disclosed embodiment, during the application phase of the interactive digital human generation model, the user's input audio to be replied to is first obtained, where the audio to be replied to can be the user's voice command or conversation content. Subsequently, the audio to be replied to is input into the pre-trained interactive digital human generation model.

[0152] The interactive digital human generation model is derived using the aforementioned training method. It can understand audio content and generate multimodal responses, including text responses, audio responses, and digital human videos. Specifically, the interactive digital human generation model analyzes the user's audio input to be replied to, generates corresponding text output as the reply content, generates audio responses that match the text content, and drives the digital human model to generate corresponding videos showing the digital human's lip movements, expressions, and movements. In this way, the interactive digital human can communicate with users in a natural and intuitive way, providing a richer and more vivid interactive experience.

[0153] The training method and interactive digital human generation model provided by the embodiments of this disclosure can be applied to the field of artificial intelligence technology. Specifically, the training method and interactive digital human generation model provided by the embodiments of this disclosure can be applied to technical fields such as speech signal processing, text-to-speech synthesis, and video generation.

[0154] The following describes the training method for the interactive digital human generation model and the generation method for the interactive digital human provided by the embodiments of the present disclosure through a specific application scenario. Figure 13 Another schematic diagram of the interactive digital human generation model provided by the embodiment of the present disclosure is shown. Figure 13 In the application scenario shown, the interactive digital human generation model is combined with the dialogue language model (corresponding to Figure 13 The "large language model" in the conversational language model and the digital human-driven model are used to generate the interactive digital human's response text, response audio, and digital human video. The digital human-driven model can be composed of an audio encoder, an image encoder, and an image decoder. The audio encoder in the digital human-driven model can be connected to the last hidden layer of the large conversational language model.

[0155] 1. Training Phase

[0156] (1) Training the Dialogue Language Model in the Interactive Digital Human Generation Model

[0157] Audio-text pairs can be used to train the speech recognition and speech synthesis capabilities of the conversational language model, while text-text pairs can be used to train the question-answering capabilities of the conversational language model.

[0158] In order to enable the conversational language model to output text responses and audio responses at the same time, during the training process of the conversational language model, the conversational language model can be controlled to alternately output N text tokens and M audio tokens.

[0159] (2) Training the Digital Human Driving Model in the Interactive Digital Human Generation Model

[0160] After the large conversational language model is trained, its parameters can be fixed and the digital human-driven model can be trained using the audio-visual synchronization training dataset. The audio-visual synchronization training dataset can include multiple audio-visual synchronization training data sets, each of which can include a pair of aligned training audio clips and target video frame sequences.

[0161] For any audio-visual synchronization training data, a training audio segment in the audio-visual synchronization training data can be input into the dialogue language model. The last hidden layer of the dialogue language model can output a first audio feature vector corresponding to the training audio segment.

[0162] The audio encoder in the digital human-driven model can encode the first audio feature vector output by the last hidden layer of the large dialogue language model to obtain a second audio feature vector. The image encoder in the digital human-driven model can encode the reference video frame to obtain a reference image feature vector. The image decoder in the digital human-driven model can decode the second audio feature vector and the reference image feature vector to obtain a synthetic video frame sequence corresponding to the training audio segment.

[0163] The pixel-level loss between the target video frame sequence and the synthesized video frame sequence can be calculated, and the parameters of the digital human driving model can be updated through back propagation based on the pixel-level loss.

[0164] 2. Application stage (prediction stage)

[0165] After the interactive digital human generation model is trained, the audio to be replied can be obtained and input into the interactive digital human generation model, which then outputs the reply text, reply audio and digital human video corresponding to the audio to be replied.

[0166] It is understood that the above-mentioned various method embodiments mentioned in this disclosure can be combined with each other to form combined embodiments without violating the principle logic. Due to space limitations, this disclosure will not go into details. It is understood by those skilled in the art that in the above-mentioned methods of specific implementation, the specific execution order of each step should be determined by its function and possible internal logic.

[0167] In addition, the present disclosure also provides a training device for generating an interactive digital human model, a generating device for an interactive digital human, a non-volatile computer-readable storage medium, and a computer program product. All of the above can be used to implement any training method for generating an interactive digital human model or a generating method for an interactive digital human provided by the present disclosure. The corresponding technical solutions and technical effects can be found in the corresponding records in the method section and will not be repeated here.

[0168] Figure 14 The block diagram of the training device for the interactive digital human generation model provided by the embodiment of the present disclosure is shown. The interactive digital human generation model includes a large dialogue language model and a digital human driving model, and the digital human driving model is connected to the target hidden layer of the large dialogue language model. Figure 14 As shown, the training device for generating an interactive digital human model includes:

[0169] An extraction module 21 is configured to input a training audio segment into the large dialogue language model and output a first audio feature vector corresponding to the training audio segment through the target hidden layer;

[0170] A generating module 22, configured to generate a synthetic video frame sequence corresponding to the training audio segment based on the first audio feature vector by using the digital human driving model;

[0171] The updating module 23 is configured to update the parameters of the digital human driving model according to the target video frame sequence and the synthesized video frame sequence corresponding to the training audio segment.

[0172] In a possible implementation, the target hidden layer is the last hidden layer of the large dialogue language model.

[0173] In a possible implementation, the digital human driving model includes an audio encoder, an image encoder, and an image decoder, and the audio encoder in the digital human driving model is connected to the target hidden layer;

[0174] The generating module 22 is used for:

[0175] Encoding the first audio feature vector by an audio encoder in the digital human driving model to obtain a second audio feature vector;

[0176] Encoding the reference video frame by the image encoder to obtain a reference image feature vector;

[0177] The second audio feature vector and the reference image feature vector are decoded by the image decoder to obtain a synthetic video frame sequence corresponding to the training audio segment.

[0178] In a possible implementation, during the parameter update process of the digital human driving model, the parameters of the large dialogue language model remain fixed.

[0179] In a possible implementation, the updating module 23 is configured to:

[0180] Calculating pixel-level loss between the target video frame sequence and the synthesized video frame sequence;

[0181] Based on the pixel-level loss, the parameters of the digital human driving model are updated through back propagation.

[0182] In a possible implementation, the apparatus further includes:

[0183] A first training module is used to perform speech recognition training and speech synthesis training on the large dialogue language model using audio-text pairs;

[0184] The second training module is used to perform question-answering training on the dialogue language model using text-text pairs.

[0185] In one possible implementation, the large conversational language model alternately outputs text tags and audio tags during training.

[0186] The present disclosure also provides a device for generating an interactive digital human, including:

[0187] The acquisition module is used to obtain the audio to be replied;

[0188] An output module, configured to input the audio to be replied to into the interactive digital human generation model, and output the reply text, reply audio and digital human video corresponding to the audio to be replied to via the interactive digital human generation model;

[0189] The interactive digital human generation model is obtained by training using the interactive digital human generation model training device.

[0190] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. Its specific implementation and technical effects can refer to the description of the above method embodiments. For the sake of brevity, they will not be repeated here.

[0191] An embodiment of the present disclosure also provides a training device for generating an interactive digital human model, comprising a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above method.

[0192] An embodiment of the present disclosure further provides a device for generating an interactive digital human, comprising a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above method.

[0193] An embodiment of the present disclosure further provides a non-volatile computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the steps of the above method when executed by a processor.

[0194] An embodiment of the present disclosure further provides a computer program product, including a computer program, or a non-volatile computer-readable storage medium carrying the computer program, wherein the computer program implements the steps of the above method when executed by a processor.

[0195] Figure 15 1 is a block diagram of a training device for generating an interactive digital human model or a generating device 1900 for generating an interactive digital human according to an exemplary embodiment. For example, the device 1900 can be provided as a server or a terminal device. Figure 15 The apparatus 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions, such as an application, that can be executed by the processing component 1922. The application stored in the memory 1932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute the instructions to perform the above-described method.

[0196] The device 1900 may also include a power supply component 1926 configured to perform power management of the device 1900, a wired or wireless network interface 1950 configured to connect the device 1900 to a network, and an input / output interface 1958 (I / O interface). The device 1900 may operate based on an operating system stored in the memory 1932, such as Windows Server 2003. TM , MacOS X TM , Unix TM , Linux TM , FreeBSD TM or similar.

[0197] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions that can be executed by the processing component 1922 of the apparatus 1900 to perform the above-described method.

[0198] A computer-readable storage medium can be a tangible device that can hold and store programs / instructions used by an instruction execution device. A computer-readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or raised structure within a groove on which instructions are stored, and any suitable combination thereof. As used herein, a computer-readable storage medium is not to be construed as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted via wires.

[0199] The computer programs (or computer-readable program instructions) described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage medium in the respective computing / processing device for storage.

[0200] The computer program (or computer program instructions) used to perform the operations of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, the state information of the computer-readable program instructions is used to personalize an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), so that the electronic circuit can execute the computer-readable program instructions, thereby implementing various aspects of the present disclosure.

[0201] Various aspects of the present disclosure are described herein with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present disclosure. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0202] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device is generated that implements the functions / actions specified in one or more blocks in the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, where these instructions cause the computer, programmable data processing device, and / or other device to operate in a specific manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing various aspects of the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0203] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0204] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and the part of the module, program segment or instruction contains one or more executable instructions for realizing the prescribed logical function. In some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the prescribed function or action, or can be implemented by a combination of dedicated hardware and computer instructions.

[0205] The computer program product may be implemented in hardware, software, or a combination thereof. In one embodiment, the computer program product is implemented as a computer storage medium. In another embodiment, the computer program product is implemented as a software product, such as a software development kit (SDK).

[0206] The above description of the various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced with each other and will not be repeated herein for the sake of brevity.

[0207] If the technical solutions of the embodiments of the present disclosure involve personal information, the products applying the technical solutions of the embodiments of the present disclosure have clearly informed the personal information processing rules and obtained the individual's voluntary consent before processing the personal information. If the technical solutions of the embodiments of the present disclosure involve sensitive personal information, the products applying the technical solutions of the embodiments of the present disclosure have obtained the individual's separate consent before processing the sensitive personal information, and at the same time meet the "explicit consent" requirement. For example, on personal information collection devices such as cameras, clear and prominent signs are set to inform that the personal information collection scope has been entered and personal information will be collected. If the individual voluntarily enters the collection scope, it is deemed that they agree to the collection of their personal information; or on the personal information processing device, when the personal information processing rules are notified by obvious signs / information, the individual's authorization is obtained through pop-up information or by asking the individual to upload their personal information. The personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the type of personal information processed.

[0208] While various embodiments of the present disclosure have been described above, the above descriptions are illustrative, non-exhaustive, and not intended to be limiting of the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, their practical applications, or improvements to existing technologies, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A training method for generating an interactive digital human model, characterized in that: The interactive digital human generation model includes a large conversation language model and a digital human driving model, and the digital human driving model is connected to the target hidden layer of the large conversation language model. The interactive digital human generation model synchronously generates output data in three modalities: text response, audio response, and digital human image based on the user's input speech. The method includes: Inputting a training audio segment into the large conversation language model, and outputting a first audio feature vector corresponding to the training audio segment through the target hidden layer; generating, by the digital human driving model, a synthetic video frame sequence corresponding to the training audio segment based on the first audio feature vector, and converting the first audio feature vector into synthetic audio by an audio decoder in the large dialogue language model; updating the parameters of the digital human driving model according to the target video frame sequence and the synthesized video frame sequence corresponding to the training audio clip; The large dialogue language model alternately outputs text tags and audio tags during training.

2. The method according to claim 1, characterized in that The target hidden layer is the last hidden layer of the large dialogue language model.

3. The method according to claim 1, characterized in that The digital human driving model includes an audio encoder, an image encoder and an image decoder, and the audio encoder in the digital human driving model is connected to the target hidden layer; Generating a synthetic video frame sequence corresponding to the training audio segment based on the first audio feature vector by using the digital human driving model includes: Encoding the first audio feature vector by an audio encoder in the digital human driving model to obtain a second audio feature vector; Encoding the reference video frame by the image encoder to obtain a reference image feature vector; The second audio feature vector and the reference image feature vector are decoded by the image decoder to obtain a synthetic video frame sequence corresponding to the training audio segment.

4. The method according to claim 1, wherein During the parameter update process of the digital human driving model, the parameters of the dialogue language model remain fixed.

5. The method according to any one of claims 1 to 4, characterized in that The updating of the parameters of the digital human driving model according to the target video frame sequence and the synthesized video frame sequence corresponding to the training audio segment includes: Calculating pixel-level loss between the target video frame sequence and the synthesized video frame sequence; Based on the pixel-level loss, the parameters of the digital human driving model are updated through back propagation.

6. The method according to claim 1, characterized in that Before inputting the training audio segment into the large dialogue language model, the method further includes: Performing speech recognition training and speech synthesis training on the large conversation language model using audio-text pairs; The large conversational language model is trained for question answering using text-to-text pairs.

7. A method for generating an interactive digital human, characterized in that: include: Get the audio to be replied; Input the audio to be replied to into the interactive digital human generation model, and output the reply text, reply audio and digital human video corresponding to the audio to be replied to through the interactive digital human generation model; Wherein, the interactive digital human generation model is trained using the interactive digital human generation model training method described in any one of claims 1 to 6.

8. A training device for interactive digital human generation model, characterized in that: The interactive digital human generation model includes a large conversation language model and a digital human driving model, and the digital human driving model is connected to the target hidden layer of the large conversation language model. The interactive digital human generation model synchronously generates output data in three modes: text response, audio response, and digital human image based on the user's input voice. The device includes: an extraction module, configured to input a training audio segment into the large dialogue language model and output a first audio feature vector corresponding to the training audio segment through the target hidden layer; a generation module, configured to generate a synthetic video frame sequence corresponding to the training audio segment based on the first audio feature vector by the digital human driving model, and convert the first audio feature vector into synthetic audio by an audio decoder in the large dialogue language model; An updating module, configured to update the parameters of the digital human driving model according to the target video frame sequence corresponding to the training audio segment and the synthesized video frame sequence; The large dialogue language model alternately outputs text tags and audio tags during training.

9. A device for generating an interactive digital human, characterized in that: include: The acquisition module is used to obtain the audio to be replied; An output module, configured to input the audio to be replied to into the interactive digital human generation model, and output the reply text, reply audio and digital human video corresponding to the audio to be replied to via the interactive digital human generation model; Wherein, the interactive digital human generation model is trained using the interactive digital human generation model training device described in claim 8.

10. A training device for interactive digital human generation model, comprising a memory, a processor, and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 6.

11. A device for generating an interactive digital human, comprising a memory, a processor, and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the method according to claim 7.

12. A non-volatile computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

13. A computer program product comprising a computer program, or a non-volatile computer-readable storage medium carrying a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images